I am trying deploy my application with AWS ECS and docker. running the environment locally on my machine with docker-compose up works. however once converted and moved to ECS I am getting error 137
I have searched the exit code and it could be ram related so I launched a larger container with 8gb of ram, still get the same issue. (there is only two images running)
I've run the images manually by connecting to the container and running "docker run -it "imagename" /bin/bash and they run without issue.
Can anyone see the cause or how I can troubleshoot this?
This is my task:
{
"requiresAttributes": [
{
"value": null,
"name": "com.amazonaws.ecs.capability.docker-remote-api.1.17",
"targetId": null,
"targetType": null
},
{
"value": null,
"name": "com.amazonaws.ecs.capability.ecr-auth",
"targetId": null,
"targetType": null
}
],
"taskDefinitionArn": "arn:aws:ecs:eu-west-1:****:task-definition/it-app:1",
"networkMode": "bridge",
"status": "ACTIVE",
"revision": 1,
"taskRoleArn": null,
"containerDefinitions": [
{
"volumesFrom": [],
"memory": 300,
"extraHosts": null,
"dnsServers": null,
"disableNetworking": null,
"dnsSearchDomains": null,
"portMappings": [],
"hostname": null,
"essential": true,
"entryPoint": null,
"mountPoints": [],
"name": "it-app",
"ulimits": null,
"dockerSecurityOptions": null,
"environment": [],
"links": null,
"workingDirectory": "/itapp/itapp",
"readonlyRootFilesystem": null,
"image": "****.dkr.ecr.eu-west-1.amazonaws.com/itapp",
"command": [
"bash",
"-c",
"python manage.py collectstatic --noinput && python manage.py makemigrations && python manage.py migrate && exec gunicorn itapp.wsgi -b 0.0.0.0:8000"
],
"user": null,
"dockerLabels": null,
"logConfiguration": null,
"cpu": 0,
"privileged": null,
"memoryReservation": null
},
{
"volumesFrom": [],
"memory": 300,
"extraHosts": null,
"dnsServers": null,
"disableNetworking": null,
"dnsSearchDomains": null,
"portMappings": [
{
"hostPort": 80,
"containerPort": 8000,
"protocol": "tcp"
}
],
"hostname": null,
"essential": true,
"entryPoint": null,
"mountPoints": [],
"name": "itapp-ngnix",
"ulimits": null,
"dockerSecurityOptions": null,
"environment": [],
"links": [
"it-app"
],
"workingDirectory": null,
"readonlyRootFilesystem": null,
"image": "****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest",
"command": null,
"user": null,
"dockerLabels": null,
"logConfiguration": null,
"cpu": 0,
"privileged": null,
"memoryReservation": null
}
],
"placementConstraints": [],
"volumes": [],
"family": "it-app"
}
EDIT:
I think I found the issue, but am not sure why I'm getting it?
service itapp-web was unable to place a task because no container instance met all of its requirements.
The closest matching container-instance 315a7d85-1e1a-4a66-9173-e1fc2c17fda2 is already using a port required by your task.
For more information, see the Troubleshooting section.
here are the container logs:
2017-09-18T15:10:07Z [INFO] Waiting for any previous stops to complete module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]" seqnum="109"
2017-09-18T15:10:07Z [INFO] Wait over; ready to move towards status: RUNNING module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]"
2017-09-18T15:10:07Z [INFO] Pulling container it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference it-app in Image State - sha256:a5a8ed72293792e2c1a51f2083d61049e9cda26b5d39a78d14087b0a2ffda675
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp in 124.116556ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]
2017-09-18T15:10:08Z [INFO] Creating container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Created container name mapping for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),] - it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING) -> ecs-it-app-1-it-app-d280ddb2fed09c640000
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Created docker container for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) -> c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64
2017-09-18T15:10:08Z [INFO] Starting container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) to CREATED, but already CREATED
2017-09-18T15:10:08Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c it-app -> RUNNING, Known Sent: NONE
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-09-18T15:10:08Z [INFO] Pulling container itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference itapp-ngnix in Image State - sha256:fb38044bb0d9b78d8da038b54d5afc1b3091b35d04524976b716500b0fcabb4f
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest in 105.243653ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:07Z [INFO] Pulling container it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference it-app in Image State - sha256:a5a8ed72293792e2c1a51f2083d61049e9cda26b5d39a78d14087b0a2ffda675
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp in 124.116556ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]
2017-09-18T15:10:08Z [INFO] Creating container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Created container name mapping for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),] - it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING) -> ecs-it-app-1-it-app-d280ddb2fed09c640000
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Created docker container for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) -> c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64
2017-09-18T15:10:08Z [INFO] Starting container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) to CREATED, but already CREATED
2017-09-18T15:10:08Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c it-app -> RUNNING, Known Sent: NONE
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-09-18T15:10:08Z [INFO] Pulling container itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference itapp-ngnix in Image State - sha256:fb38044bb0d9b78d8da038b54d5afc1b3091b35d04524976b716500b0fcabb4f
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest in 105.243653ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:08Z [INFO] Creating container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]" container="itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [WARN] Warning, link with no linkalias module="api client" linkName="it-app" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]" container="itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Created container name mapping for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),] - itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (PULLED->RUNNING) -> ecs-it-app-1-itapp-ngnix-c4e29c838bb19df6cb01
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:09Z [INFO] Created docker container for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (CREATED->RUNNING) Containers: [itapp-ngnix (CREATED->RUNNING),it-app (RUNNING->RUNNING),]: itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (CREATED->RUNNING) -> d06b6e4300665f0f8c18f0be04345e4797827736574d977b2a81cfb5d7eb471a
2017-09-18T15:10:09Z [INFO] Starting container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (CREATED->RUNNING) Containers: [itapp-ngnix (CREATED->RUNNING),it-app (RUNNING->RUNNING),]" container="itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (CREATED->RUNNING)"
2017-09-18T15:10:09Z [INFO] Task change event module="TaskEngine" event="{TaskArn:arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c Status:RUNNING Reason: Containers:[] Task:it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (RUNNING->RUNNING) Containers: [itapp-ngnix (RUNNING->RUNNING),it-app (RUNNING->RUNNING),]}"
2017-09-18T15:10:09Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c itapp-ngnix -> RUNNING, Ports [{8000 80 0.0.0.0 0}], Known Sent: NONE
2017-09-18T15:10:09Z [INFO] TaskHandler, Adding event: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> RUNNING, Known Sent: NONE
2017-09-18T15:10:09Z [INFO] TaskHandler, Sending task change: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> RUNNING, Known Sent: NONE
2017-09-18T15:10:09Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (RUNNING->RUNNING) Containers: [itapp-ngnix (RUNNING->RUNNING),it-app (RUNNING->RUNNING),]: itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-09-18T15:10:09Z [INFO] Stopping container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (RUNNING->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (RUNNING->STOPPED),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (RUNNING->STOPPED)"
2017-09-18T15:10:09Z [INFO] Error retrieving stats for container d06b6e4300665f0f8c18f0be04345e4797827736574d977b2a81cfb5d7eb471a: context canceled
2017-09-18T15:10:09Z [INFO] Container d06b6e4300665f0f8c18f0be04345e4797827736574d977b2a81cfb5d7eb471a is terminal, stopping stats collection
2017-09-18T15:10:09Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c itapp-ngnix -> STOPPED, Exit 1, , Known Sent: NONE
2017-09-18T15:10:18Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:40Z [WARN] Error converting stats for container c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64: Invalid container statistics reported
2017-09-18T15:10:40Z [INFO] Task change event module="TaskEngine" event="{TaskArn:arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c Status:STOPPED Reason: Containers:[] Task:it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (STOPPED->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (STOPPED->STOPPED),]}"
2017-09-18T15:10:40Z [INFO] Error retrieving stats for container c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64: context canceled
2017-09-18T15:10:40Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c it-app -> STOPPED, Exit 137, , Known Sent: NONE
2017-09-18T15:10:40Z [INFO] TaskHandler, Adding event: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> STOPPED, Known Sent: RUNNING
2017-09-18T15:10:40Z [INFO] TaskHandler, Sending task change: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> STOPPED, Known Sent: RUNNING
2017-09-18T15:10:40Z [INFO] Container c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64 is terminal, stopping stats collection
2017-09-18T15:10:40Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (STOPPED->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (STOPPED->STOPPED),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (STOPPED->STOPPED) - Exit: 137 to STOPPED, but already STOPPED
2017-09-18T15:10:40Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:40Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (STOPPED->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (STOPPED->STOPPED),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (STOPPED->STOPPED) - Exit: 137 to STOPPED, but already STOPPED
2017-09-18T15:10:50Z [INFO] Saving state! module="statemanager"
I ended up deleting my service and starting again and now it is running