I have an Autoscaling group, with EC2 instances that are manage via lifecycle hooks. One thing the lifecycle hooks do, is some cleanup on the EC2 Instances ensuring all processes are stopped properly. For debugging purposes, this cleanup is logged to log files that are send to cloudwatch logs via cloudwatch log agent.
The problem is, that when the lifecycle hook completes, the instance is terminated and the cloudwatch agent might not yet have uploaded all the logs.
Is there a way to tell cloudwatch agent to send logs now! Or wait for it to update the logs?
I think there are 2 issues here: 1- for the agent to have enough time to gracefully stop 2- for the agent to flush all cached data
The first problem, is explained in this AWS Blog post: https://aws.amazon.com/blogs/infrastructure-and-automation/run-code-before-terminating-an-ec2-auto-scaling-instance/ which essentially is using a feature called "lifecycle hooks" and run something as response to these hook events.
The second problem has an open issue and is suggested to send SIGTERM and it "should" flush before shutting down the process. https://github.com/aws/amazon-cloudwatch-agent/issues/170