My current setup:
I am using Docker to actually build the app on the instance. After the repository is cloned onto the instance, I build the docker image, inside of which I run my gunicorn server, and I forward port 80 requests to the image.
My question is regarding "collectstatic". Right now, I include "python manage.py collectstatic" in my Dockerfile as one of the build steps.
The problem I am facing is that collectstatic is currently run on each new instance during deployment, thus duplicating the work. It also results in collisions when new static files need to be uploaded to s3, or modified static files need to be updated. One instance will delete the file and begin copying it over, and the second instance will try to perform a read operation on a file that just got deleted, and will crash with botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found
How should I be managing updating static files during deployment? I would like to be able to run collectstatic only once during the deployment, but not sure how to integrate that with my current deployment via CodeDeploy, which does not seem to offer any pre-deployment hooks where I could run a custom script (CodeDeploy has per-instance hooks, but this would not help as I need something that happens on a given deployment, rather than on each instance created during the deployment)
Any help or advice would be appreciated. I am sure that there is something wrong here with respect to my setup, so any information or guidance on how static files are typically handled during deployment (when you have multiple instances you're deploying to) would be super helpful.
I think your problem that you are building the docker image on all the target EC2 instances.
A better CI/CD flow will be that you build the docker image once then push the image to a container registry e.g. ECR. Also push any static files updates to S3.
Finally, to deploy on the target EC2 instances, you just pull the docker image and run it.