Search code examples
amazon-web-servicesnginxamazon-ec2amazon-cloudwatch

NGINX logs not showing in CloudWatch

Thank you in advance for the help. I'm quite new in AWS.

I have tried to log the NGINX logs from my EC2 instance to CloudWatch but I was not able to make it work and I don’t know what am I missing.

In summary the steps I followed are:

  1. As I am running NGINX in a docker I mapped the volume to my local folder (let’s say /home/ubuntu/nginx/logs/) and I checked that it is logging into the files access.log error.log json_access.log
  2. I installed the CloudWatch Agent following this: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html
  3. I created an IAM role with CloudWatchAgentServerPolicy policy.
  4. I added that IAM role to the EC2 instance running my NGINX
  5. I configured the CloudWatch Agent with sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard giving the following answers:
    1. On which OS are you planning to use the agent? Answer: linux
    2. Are you using EC2 or On-Premises hosts? Answer: EC2
    3. Which user are you planning to run the agent? Answer: root
    4. Do you want to turn on StatsD daemon? Answer: NO
    5. Do you want to monitor metrics from CollectD? Answer: NO
    6. Do you want to monitor any host metrics? Answer: NO
    7. Do you have any existing CloudWatch Log Agent? Answer: NO
    8. Do you want to monitor any log files? Answer: YES
    9. Log file path: /home/ubuntu/nginx/logs/
    10. Log group name: nginx
    11. Log stream name: [{instance_id}] (default)
    12. Do you want to specify any additional log files to monitor? Answer: NO
    13. Do you want to store the config in the SSM parameter store? Answer: NO
  6. I started the agent: sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a start

With these steps I would have expected that the agent would automatically push the log group and stream into cloudwatch and I would have all the raw logs available in CloudWatch > Log groups > nginx > myinstanceid.

But that didn’t happen.

So I manually created the log group and the log stream by naming them the same as in the config and still nothing. I also tried redoing everything with a explicit log stream name just in case I was spelling the instance id incorrectly, but nothing.

This is the config.json generated from the wizard:

{
    "agent": {
        "run_as_user": "root"
    },
    "logs": {
        "logs_collected": {
            "files": {
                "collect_list": [
                    {
                        "file_path": "/home/ubuntu/nginx/logs/",
                        "log_group_name": "nginx",
                        "log_stream_name": "{instance_id}",
                        "retention_in_days": 30
                    }
                ]
            }
        }
    }

What am I missing? Or what am I doing wrong?

Thank you so much again!

EDIT: output of the agent logs /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log:

(Note: I had to copy manually the config.json from the wizard into /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json.)

2023/04/19 06:46:52 I! D! [EC2] Found active network interface
I! Detected the instance is EC2
2023/04/19 06:46:52 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2023/04/19 06:46:52 I! Valid Json input schema.
I! Detecting run_as_user...
I! Trying to detect region from ec2
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
 
2023/04/19 06:46:52 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml 
2023/04/19 06:46:52 D! toml config [agent]
  collection_jitter = "0s"
  debug = false
  flush_interval = "1s"
  flush_jitter = "0s"
  hostname = ""
  interval = "60s"
  logfile = "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"
  logtarget = "lumberjack"
  metric_batch_size = 1000
  metric_buffer_limit = 10000
  omit_hostname = false
  precision = ""
  quiet = false
  round_interval = false

[inputs]

  [[inputs.logfile]]
    destination = "cloudwatchlogs"
    file_state_folder = "/opt/aws/amazon-cloudwatch-agent/logs/state"

    [[inputs.logfile.file_config]]
      file_path = "/home/ubuntu/nginx/logs/"
      from_beginning = true
      log_group_name = "nginx"
      log_stream_name = "MY INSTANCE ID"
      pipe = false
      retention_in_days = 30
    [inputs.logfile.tags]
      metricPath = "logs"

[outputs]

  [[outputs.cloudwatchlogs]]
    force_flush_interval = "5s"
    log_stream_name = "MY INSTANCE ID"
    region = "eu-west-1"
    tagexclude = ["metricPath"]
    [outputs.cloudwatchlogs.tagpass]
      metricPath = ["logs"]
2023/04/19 06:46:52 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json ...
2023/04/19 06:46:52 I! Valid Json input schema.
2023/04/19 06:46:52 I! Detected runAsUser: root
2023/04/19 06:46:52 I! Changing ownership of [/opt/aws/amazon-cloudwatch-agent/logs /opt/aws/amazon-cloudwatch-agent/etc /opt/aws/amazon-cloudwatch-agent/var] to 0:0
2023-04-19T06:46:53Z I! Starting AmazonCloudWatchAgent 1.247358.0
2023-04-19T06:46:53Z I! AWS SDK log level not set
2023-04-19T06:46:53Z I! Loaded inputs: logfile
2023-04-19T06:46:53Z I! Loaded aggregators: 
2023-04-19T06:46:53Z I! Loaded processors: 
2023-04-19T06:46:53Z I! Loaded outputs: cloudwatchlogs
2023-04-19T06:46:53Z I! Tags enabled: host=ip-172-XX-XX-XXX
2023-04-19T06:46:53Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"ip-172-XX-XX-XXX", Flush Interval:1s
2023-04-19T06:46:53Z I! [logagent] starting
2023-04-19T06:46:53Z I! [logagent] found plugin cloudwatchlogs is a log backend
2023-04-19T06:46:53Z I! [logagent] found plugin logfile is a log collection

Also:

$ sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
{
  "status": "running",
  "starttime": "2023-04-19T06:46:52+00:00",
  "configstatus": "configured",
  "version": "1.247358.0b252413"
}
Solution

  • You can refer to my code snippet:

    sudo wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb -O /tmp/amazon-cloudwatch-agent.deb 
sudo dpkg -i /tmp/amazon-cloudwatch-agent.deb 
 
sudo chmod +x /home/ubuntu/nginx/logs 
sudo tee -a /opt/aws/amazon-cloudwatch-agent/bin/config.json << EOF 
{ 
   "agent": { 
      "run_as_user": "root" 
   }, 
   "metrics":{ 
      "namespace":"CwAgent", 
      "append_dimensions": { 
          "InstanceId": "\${aws:InstanceId}" 
      }, 
      "aggregation_dimensions" : [["InstanceId"]], 
      "metrics_collected":{ 
         "mem":{ 
            "measurement":[ 
               "mem_used_percent" 
            ] 
         }, 
         "disk":{ 
            "measurement":[ 
               "used_percent" 
            ], 
            "resources":[ 
               "/" 
            ] 
         } 
      } 
   }, 
   "logs":{ 
      "logs_collected":{ 
         "files":{ 
            "collect_list":[ 
               { 
                 "file_path": "/home/ubuntu/nginx/logs/access*", 
                 "log_group_name": "/api", 
                 "log_stream_name": "{instance_id}" 
               },
               { 
                 "file_path": "/home/ubuntu/nginx/logs/error*", 
                 "log_group_name": "/api", 
                 "log_stream_name": "{instance_id}" 
               }
            ] 
         } 
      } 
   } 
} 
EOF 
 
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -s 

sudo amazon-cloudwatch-agent-ctl -a status