Aws x-ray doesn't work for a sidecar container on fargate

Question

I tried to set up aws x-ray on fargatge. But there is no errors and metrics do not appear in x-ray. Here is part of my task definition for fargate. I tried to debug potential issue, but on local machine using docker compose and x-ray daemon it works, what suggest that application is configured properly.

demoRole - has full access to xray

security group - is open for whole world on TCP 80,2000 and UDP 2000

in that particular task definition I added environment variables. But removing them change nothing.

{
  "executionRoleArn": "arn:aws:iam::************:role/demoRole",
  "containerDefinitions": [
    {
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "ecs/demo",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 80
        }
      ],
      "cpu": 224,
      "environment": [
        {
          "name": "AWS_XRAY_DAEMON_ADDRESS",
          "value": "0.0.0.0:2000"
        }
      ],
      "memory": 384,
      "image": "************.dkr.ecr.us-east-1.amazonaws.com/demo:06031131",
      "name": "demo"
    },
    {
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "ecs/demo",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "portMappings": [
        {
          "hostPort": 2000,
          "protocol": "udp",
          "containerPort": 2000
        }
      ],
      "cpu": 32,
      "environment": [
        {
          "name": "AWS_REGION",
          "value": "us-east-1"
        }
      ],
      "memory": 128,
      "image": "amazon/aws-xray-daemon",
      "name": "xray-sidecar"
    }
  ],
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "256",
  "status": "ACTIVE",
}

These are logs from both containers:

x-ray

2020-03-11 13:45:19caused by: Get http://169.254.169.254/latest/meta-data/instance-id: dial tcp 169.254.169.254:80: connect: invalid argument
2020-03-11 13:45:192020-03-11T12:45:19Z [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.us-east-1.amazonaws.com
2020-03-11 13:45:192020-03-11T12:45:19Z [Info] Starting proxy http server on 0.0.0.0:2000
2020-03-11 13:45:182020-03-11T12:45:18Z [Info] Using region: us-east-1
2020-03-11 13:45:182020-03-11T12:45:18Z [Info] Initializing AWS X-Ray daemon 3.2.0
2020-03-11 13:45:182020-03-11T12:45:18Z [Info] Using buffer memory limit of 39 MB
2020-03-11 13:45:182020-03-11T12:45:18Z [Info] 624 segment buffers allocated

service:

2020-03-11 14:18:42- - [11/Mar/2020:13:18:42 +0000] "GET /demo/health_check HTTP/1.0" 200 2 "-" "ELB-HealthChecker/2.0"
2020-03-11 14:18:42[2020-03-11 13:18:42 +0000] [27] [DEBUG] GET /demo/health_check
2020-03-11 14:18:42[2020-03-11 13:18:42 +0000] [25] [DEBUG] GET /demo/health_check
2020-03-11 14:18:35[2020-03-11 13:18:35,721] INFO in middleware: initializing xray middleware

In other test i tried to set up separate task for x-ray, still doesn't work. Is there something I missing?

Answer 1

Can you enable the X-Ray SDK's DEBUG MODE on your service you are instrumenting? It looks like the configs are correct, it may be a problem with your instrumentation setup. This will enable additional logs so we can diagnose the issue.