How to keep logs for ECS container?

Question

I'm running ECS tasks, and recently the service cpu hit 100% and went down.

I waited for the instance to settle down and sshed-in. I was looking for logs, but it seemed docker container restarted and logs are all gone (logs when the cpu was high)

Next time, how do I make sure I can see the logs at least to diagnose the problem?

I have the following, hoping to see some logs somewhere (mounted in the host machine)

     "mountPoints": [
            {
                "readOnly": null,
                "containerPath": "/var/log/uwsgi",
                "sourceVolume": "logs"
            }
        ],

But there's no /var/log/uwsgi in the host machine. And I probably need syslog and stuff..

Answer 1

As far you current configuration logs totally depend on the path that you define in the volume section.

     "mountPoints": [
            {
                "readOnly": null,
                "containerPath": "/var/log/uwsgi",
                "sourceVolume": "logs"
            }
        ],

the souces path defined in volume logs logs not /var/log/uwsgi, so you are mounting

/var/log/uwsgi (container path) -> logs volume (host path). you find these logs in path define in logs volume. but better to set something like

            {
                "readOnly": null,
                "containerPath": "/var/log/uwsgi",
                "sourceVolume": "volume_name"
            }

then volume config

"volumes": [
    {
      "name": "logs",
      "host": {
        "sourcePath": "/home/ec2-user/logs"
      }
    }
]

From documentation

In the task definition volumes section, define a bind mount with name and sourcePath values.

  "volumes": [
    {
      "name": "webdata",
      "host": {
        "sourcePath": "/ecs/webdata"
      }
    }
  ]

In the containerDefinitions section, define a container with mountPoints values that reference the name of the defined bind mount and the containerPath value to mount the bind mount at on the container.

  "containerDefinitions": [
    {
      "name": "web",
      "image": "nginx",
      "cpu": 99,
      "memory": 100,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80
        }
      ],
      "essential": true,
      "mountPoints": [
        {
          "sourceVolume": "webdata",
          "containerPath": "/usr/share/nginx/html"
        }
      ]
    }
  ]

bind-mounts-ECS

Now if I come to my suggestion I will go for AWS log driver.

Working in AWS, the best approach is to push all logs to CW, but AWS log driver only pushes container stdout and stderr logs to CW.

Using AWS log driver you do not need to worry about instance and container, you will log in CW and you can stream these logs to ELK as well.

          "logConfiguration": {
                "logDriver": "awslogs",
                "options": {
                    "awslogs-group": "awslogs-wordpress",
                    "awslogs-region": "us-west-2",
                    "awslogs-stream-prefix": "awslogs-example"
                }
            }

using_awslogs