I have an app running in a custom Java container on App Engine flexible. I've been trying to track down why I've been getting stale responses, and I narrowed the issue down to responses being cached.
I know our app isn't returning the stale content, because we have no shared cache, and stale responses persist after we redeploy. We also aren't using Cloud CDN.
When I hit my app's API at https://[app-id].appspot.com/api/123, my app returns a Cache-Control: public, s-maxage=31536000, max-age=3600 response header. s-maxage was intended to tell Cloudflare (our CDN) to cache the response, and we manually purge the Cloudflare's cache when we update content. However, it appears that something in front of our app is caching responses. In the cached responses, the date response header stays the same and the age increases, showing that the content is indeed stale.
I've seen documentation regarding caching in the App Engine standard docs for app.yaml, but I couldn't find any mention of caching infrastructure in the App Engine flexible docs. My mental model was that only HTTPS load balancing was happening on behalf of the app, but that doesn't appear to be the case.
If I try to reproduce this issue from within GCP by SSHing into a Compute Engine instance or App Engine Flexible instance, I don't observe any caching. However, I've tried this from other clouds and laptop and the caching is consistent. Cloudflare support was who originally alerted us to this behavior, and they're seeing the caching from their production data centers, which are peered directly with Google via their CDN Interconnect program.
This must be coming from Google's infrastructure itself, any maybe the hidden Google Cloud Load Balancer used for App Engine, since the connection is encrypted.
What's doing the caching, how do I disable it, and how do I purge it?
The serving infrastructure in front of your App Engine instances can potentially cache requests though this is not an officially documented feature. You're more likely to see this if you have a lot of clients hitting App Engine from the same network, resulting in all of them hitting the same group of caches and keeping those caches warm. There is no API to flush the cache so the suggestion provided in the comments above by John Hanley is the best mitigation: use version-specific URLs for your static files that you only want to be served by a specific version of your app.