Cloudfront is caching 404s from nginx for assets that do exist on the origin server

Question

I've been tinkering with setting up multiple cloudfront endpoints for a rails app running behind nginx to improve page load times. Basically - we had a single endpoint set up before that appears to have been working fine, but when I added in a second endpoint using the following declaration for asset_host:

config.action_controller.asset_host = Proc.new do |source|
    hosts = ["https://url1.cloudfront.net", "https://url2.cloudfront.net"]
    hosts[source.hash % 2]
end

whenever I deploy (using a pretty vanilla capistrano deploy script), some assets are not loading - cloudfront is caching the nginx 404 page. If I invalidate cloudfront's cache, the assets all load fine.

The capistrano script itself does the compilation before it ever restarts unicorn, so there should be no html being served with references to the new asset file names, yet nevertheless, cloudfront is caching 404s immediately after the deploy.

I certainly can't invalidate the cloudfront caches after every deploy, that takes too long. Has anyone encountered this issue? Any suggestions for how to work around this?

Answer 1

I figured this out. Turns out our preload and asset change monitoring endpoint (which reports to the front end when assets change and need to be reloaded) was testing against the on disk list of digests to make this determination. Naturally - the on disk digest could get ahead of which digests had actually been compiled across all the machines, leading browsers to attempt to fetch assets before they were actually ready.

For others using a technique like this to test for changes in assets - might I recommend using the hash in the application stored at:

MyAppNamespace::Application.config.assets.digests

Hope this helps someone else!

[UPDATE] Actually - the real source of the problem was using the :hash method to determine which url to serve - while the output of that method will be consistent within a single process - it will not be across processes, thus different servers were serving different hashes, and since they're all behind a balancer, not all servers had the assets being requested.