I swapped from Alpine 3.16 to Debian Bullseye slim with jemalloc and somehow this change has broken the redirect based variant URLs for ActiveStorage.
The method I'm using to generate the URLs is url_for( image.variant(resize_to_limit: [1200, nil]) ) and then I embed that URL into a document stored in the DB. This produces a URL that looks like this:
https://subdomain.example.com/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBdjRDIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--88d725d3131d3979e984fb64b7503ad8dba711c2/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaDdCem9MWm05eWJXRjBTU0lJY0c1bkJqb0dSVlE2QzNKbGMybDZaVWtpQ2pFeU1EQStCanNHVkE9PSIsImV4cCI6bnVsbCwicHVyIjoidmFyaWF0aW9uIn19--74294a2119561b1929753c5929e8d0db9fcc9325/Screen%20Shot%202021-06-17%20at%205.41.25%20PM.png
After swapping OS to Debian Bullseye, the old URLs 404. It's a browser default 404 and not a Rails or app custom 404. If I generate the URL with the same method on the same file, it looks like this:
https://subdomain.example.com/rails/active_storage/representations/redirect/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBBdjRDIiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--3b52e0fc8a157c21903976818d9dd26ddc11debe/eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaDdCem9MWm05eWJXRjBTU0lJY0c1bkJqb0dSVlE2RkhKbGMybDZaVjkwYjE5c2FXMXBkRnNIYVFLd0JEQT0iLCJleHAiOm51bGwsInB1ciI6InZhcmlhdGlvbiJ9fQ==--6b3587a792240d45bcec733c6cf847d3d371b58f/Screen%20Shot%202021-06-17%20at%205.41.25%20PM.png
Any ideas why swapping out the OS would do this?
Update:
Upon rolling back to Alpine, the images work again, however the generated URLs are still different. I believe portions of the URL have do to with expiry. So the question isn't why the URLs change but why would an already generated URL become invalid with a new OS? Perhaps there's some hashing going on which changes when a low level library is changed?
If you base64 decode the hash looking part of the URL part of the decode looks like this:
{"_rails":{"message":"BAh7BzoLZm9ybWF0SSIIcG5nBjoGRVQ6FHJlc2l6ZV90b19saW1pdFsHaQKwBDA=","exp":null,"pur":"variation"}}
Both URLs have the same json and message - it's the other stuff around it that seems to be causing the difference and the 404.
The local logs for the 404 look like this:
Processing by ActiveStorage::Representations::RedirectController#show as */*
Parameters: {"signed_blob_id"=>"eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaHBDUT09IiwiZXhwIjpudWxsLCJwdXIiOiJibG9iX2lkIn19--78168cd86530b23b5c3f81cc8a2818da8a661df9", "variation_key"=>"eyJfcmFpbHMiOnsibWVzc2FnZSI6IkJBaDdCem9MWm05eWJXRjBTU0lJU2xCSEJqb0dSVlE2RkhKbGMybDZaVjkwYjE5c2FXMXBkRnNIYVFLd0JEQT0iLCJleHAiOm51bGwsInB1ciI6InZhcmlhdGlvbiJ9fQ==--54c153007774686020b21201cc609413b0ad9779", "filename"=>"IMG_6361"}
Filter chain halted as :set_blob rendered or redirected
Completed 404 Not Found in 1ms (ActiveRecord: 0.0ms | Allocations: 724)
This is bonkers but I've found the issue. Between the two images I changed the ENV["SECRET_KEY_BASE"] which is only set in the Dockerfile to allow precompiling production assets into the image. A real SECRET_KEY_BASE is later injected by Dokku in production.
Even though that env var was only used temporarily, it seems to cause previously generated variant URLs to 404. I haven't a clue (yet) why that variable should affect variant URLs. That means if you ever want to roll that variable, which is key for security, you would have to migrate all <img> tags in user submitted content.
If you run into this issue, ensure your SECRET_KEY_BASE is the same across all images. If you need to roll that variable, then... god speed.