AWS beanstalk monitoring metrics disappear during deploy

Question

We have setup an immutable configuration for AWS beanstalk using both web and worker classes (multiple clusters). When we deploy a new application, it creates a temporary autoscaling group and then deploys to that and in end switches back to the old autoscaling group. This process takes about 20-30 mins and works fine.

Although, every time we deploy the application the monitoring stats: CPU utilization, Memory utilization, disk space and so on, all disappear for 5-6 hours before returning back. Seems like an AWS issue but not sure if we are doing anything wrong. Has anyone else experience such behavior? Is there a workaround?

EDIT: Here's the screenshot for missing memory metric, as you can see there was no data between 5.30pm and 11.30pm.

Answer 1

For anyone who finds this issue in future - I figured out the issue. The script I am using is provided by Cloudwatch that internally caches the auto-scaling group name for 6 hours. Typically that would not be an issue, since these values do not change regularly. But due to immutable deploy a temporary auto-scaling group is created during deploy that gets cached for 6 hours.

To resolve this, I replaced the following line in the CloudWatchClient code:

$meta_data_short_ttl = 21600; # 6 hours

with:

meta_data_short_ttl = 600; # 10 mins

Can also be done via .ebextension change from existing code if you are using this script:

04-reduce-cache:
    command: sed -i 's/meta_data_short_ttl = 21600; # 6 hours/meta_data_short_ttl = 600; # 10 mins/g' /opt/cloudwatch/aws-scripts-mon/CloudWatchClient.pm