I am deploying services to ECS fargate behind ALB. During deployment, ALB sends health check to the service and if there are 3 consecutive failure health checks, ECS will destroy the new deployed service and keep the old version of the container. I am looking for a way to monitor the deployment failure cases. One possible solution is to monitor the ECS task status change. Send an alert if the container status becomes STOP. But this solution is not specific to deployment. The container can become STOPPED anytime if there is an error. Also during deployment, the old container's status will become STOPPED as well. So is there any other metrics I can use to monitor the deployment failure?
Normally, we will integrate the deployment checking at the end of our CI/CD systems.
I am not sure about which CI tool you are using, but if you used Jenkins, you can do that on the post stage.
And after you update the ECS Service, there is a Deployments label on the ECS Service console, you can check there until the ACTIVE row disappeared. That means the new task has been deployed. It also works on the aws-cli, so you can use aws-cli and jq to run a simple loop to check if your new task deployed.
I have a sample script below can be a reference
#!/bin/bash
RESULT=$(aws ecs describe-services --cluster ${ECS_CLUSTER} --service ${SERVICE_NAME} \
| jq -r '.services[].deployments[] | select(.status == "ACTIVE")')
# No ACTIVE status means deployment complete
if [ "$RESULT" = "" ]; then
exit 0
else
echo "$RESULT"
exit 1
fi
Hopes it help you.