Cloudwatch agent not sending logs to cloudwatch

Question

I am trying to send two lots of logs up to CloudWatch.

Here are the two logs:

1. /var/log/apache2/access.log
2. /var/log/apache2/error.log

I used the amazon-cloudwatch-agent-config-wizard to create the config file, and here is a snippet of the file showing the correct file path:

"collect_list": [
    {
         "file_path": "/var/log/apache2/access.log",
         "log_group_name": "*group_name*",
         "log_stream_name": "apache-access"
    },
    {
         "file_path": "/var/log/apache2/error.log",
         "log_group_name": "group-name*",
         "log_stream_name": "apache-error"
    }
]

I loaded in the config with:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

And ran into no errors and no errors and showing in the amazon-cloudwatch-agent.log.

Checking the status of the Amazon CloudWatch Agent shows it is running and has not errors. Also states the schema is valid.

The unique part of all of this is that I have removed the old CloudWatch agent and installed the new one. I have done this on two EC2 instances, one of which everything is working perfectly on, and the other one is not sending the logs to CloudWatch.

In a nutshell, why aren't the logs going up to CloudWatch? What can I do to troubleshoot this?

Any help will be appreciated.

Answer 1

So the problem turned out to be permission-based. The CloudWatch config wizard defaults to using cwagent as the user that runs CloudWatch, this is also reiterated in official guides.

Changing the running using to root resolved the issue even though the files in question all had 777 permissions at the time of trying to get it running.

The config file you edit is:

sudo nano /opt/aws/amazon-cloudwatch-agent/bin/config.json

At the top of the file you will see:

"agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "cwagent"
},

You need to change run_as_user to root, like:

"agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root"
},

Once you have changed that, you simply reload the config file:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s

And then restart the service:

sudo systemctl restart amazon-cloudwatch-agent.service

You should then see the logs coming into CloudWatch. Expect some backfilling.