My Varnish server caches a maps tile server, which is updated real-time from OpenStreetMap every 1 minute. Frequently, an entire area of the map needs to be invalidated -- i.e. 10,000 or even 100,000 tiles at once. Each tile is a single URL (no variances).
Is there an efficient way to run such large scale Varnish invalidation? Ideally objects should remain in cache (so that grace period would continue to work unless a URL flag of nograce is passed in), but marked as no longer valid. Ideally this tight loop would be implemented in VCL itself.
Example URL: http://example.org/tile/10/140/11.pbf (no variance, no query portion) where the numbers are {zoom}/{x}/{y}, and the list of those numbers (i.e. 100,000 at a time) is generated externally every minute and stored in a file. BTW, most likely most of those URLs won't even be in cache.
The answer depends a lot on how those URLs look like. Options are:
Using multiple soft purges [1] (beware of the 'soft' part; you'll need the purge VMOD for that) triggered by an external loop (sorry, you cannot do that in VCL). Soft purges set TTL to 0 instead of fully removing objects from the storage.
Using a simple ban [2]. However, bans will completely (and lazily) remove matching objects from the storage (i.e. there is not 'soft' flavour for bans).
Using the xkey VMOD [3]. The VMOD provides a 'soft' invalidation option, but not sure if a surrogate index would help for your use case.
[1] https://varnish-cache.org/docs/trunk/reference/vmod_purge.html
[2] https://varnish-cache.org/docs/trunk/users-guide/purging.html#bans
[3] https://github.com/varnish/varnish-modules/blob/master/docs/vmod_xkey.rst