AWS Elastic Beanstalk - /root/.docker/config.json not updated with new credentials when ECR has been used

Question

I'm currently switching from ECR to GitHub packages for hosting of our docker images. After a lot of trial and error, and a lot of messages like no basic auth credentials when pulling the images, I started digging deeper by SSH:ing into the beanstalk instance to investigate the actual files. It turns out that when I update the .dockercfg in my S3 bucket with new credentials, these are copied as expected to /root/.dockercfg on the beanstalk, but aren't reflected in /root/.docker/config.json. This file only contains the ECR credentials, not the GitHub ones. If I manually add the GitHub credentials, I can pull the images just fine. It seems like /root/.docker/config.json overrides the credentials in /root/.dockercfg, which only exist during the deploys.

How can I solve this?

Edit:

The root cause seems to be that /root/.docker/config.json is created when ECR is used, and is not updated by other private registry authentications, since ECR is treated different. When I set up a new beanstalk that uses GitHub from the beginning, everything works as expected. This part of the /opt/elasticbeanstalk/hooks/pre/03build.sh creates the file:

# if the image is in an ECR repo, authenticate with ECR
ECR_IMAGE_PATTERN="^([a-zA-Z0-9][a-zA-Z0-9_-]*)\\.dkr\\.ecr\\.([a-zA-Z0-9][a-zA-Z0-9_-]*)\\.amazonaws\\.com(\\.cn)?/.*"
if [[ $FROM_IMAGE =~ $ECR_IMAGE_PATTERN ]]; then
        ECR_REGISTRY_ID=${BASH_REMATCH[1]}
        ECR_REGION=${BASH_REMATCH[2]}

        ECR_LOGIN_RESPONSE=`aws ecr get-login --no-include-email --registry-ids $ECR_REGISTRY_ID --region $ECR_REGION 2>&1`
        [ $? -eq 0 ] || error_exit "Failed to authenticate with ECR for registry '$ECR_REGISTRY_ID' in '$ECR_REGION'" 1

        # output of aws ecr get-login should be a "docker login" command, simply invoke it
        echo $ECR_LOGIN_RESPONSE | grep -q "^docker login" || error_exit "Invalid response from 'aws ecr get-login', expecting a 'docker login' command, was: '$ECR_LOGIN_RESPONSE'."
        eval $ECR_LOGIN_RESPONSE
fi

Answer 1

A workaround is to delete /root/.docker/config.json or to enable immutable deploys so that new EC2 instances are created.