Search code examples
amazon-ec2amazon-cloudfrontcapifony

How do I ensure Cloudfront has the correct asset version when doing a rolling deployment?

We're currently using Capifony and the ec2-capify plugin to do rolling deployments of our code to a set of instances behind an ELB. We also use CloudFront to manage static assets, which we version with a query string (e.g. ?v1 or ?v2).

We've stumbled across a rare issue regarding updating the asset version. If the current version is v1, and we do a rolling deployment of v2 one server at a time then the following can happen for requests on the v2 box:

  1. CloudFront is asked for v2 and misses.
  2. CloudFront goes to the ELB and asks for the asset.
  3. The ELB chooses a server and one of two things happens: Cloudfront hits one of the new deployed servers (serving v2) OR it hits the old servers (v1).
  4. Either way, Cloudfront stores the content as v2. In the case where it hit a v1 server, the content is served incorrectly.

Our current solution to this is we have to do another deploy with a new asset version.

Is there a way to force Cloudfront through the ELB to only hit one of our updated (v2) servers, and to ignore the v1's?

Or am I missing an alternate solution which would resolve the problem?

Solution

  • The approach we opted for was to broadly abandon our existing asset deployment pipeline. In the "old" way, we opted for an asset.css?v=<version> model, where CloudFront was pointed at an origin which was served by multiple instances.

    The way we resolved it was to move to a hash name asset model and S3 based origin. This means that instead of asset.css?v=<version> we have asset-<hash-of-contents>.css which are synced to an S3 bucket. The bucket progressively adds newer and newer versions, but old versions are always available if we decided to go back or if something like an email links to it (common problem with images).

    The sync to S3 script runs before we deploy to our webservers that contain the HTML that references the assets, so CloudFront is always able to serve the latest asset.

    Here's a sample script:

    #!/usr/bin/env bash

set -e # Fail on any error
set -x

if [ "$#" -ne "1" ]
then
    echo "Usage: call with the name of the environment you're deploying to"
    exit 1
fi

CDN_ENVIRONMENT=$1

S3_BUCKET="s3://static-bucket-name-of-your-choice/${CDN_ENVIRONMENT}/"

echo "Generating assets

... do the asset generation here ...

echo "Copying to S3"

# Now do the actual copying of the web dir. We use size-only because otherwise all files are newer, and all get copied.
aws s3 sync --exclude "some-folder-to-exclude/*" --acl public-read --size-only ./web/ ${S3_BUCKET}

echo "Copy to S3 complete"