Search code examples
phpamazon-ecsstracefpm

Strace utility does not work inside AWS ECS task

we have some php-fpm processes getting stuck in our production system on ECS FARGATE tasks. I tried debugging those services using: "strace -p 50", where 50 is PID for my php-fpm child worker process and I am running this under root user, however getting this error:

Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf

I have modified the setting in /etc/sysctl.d/10-ptrace.conf and saved the file, however I am still getting the same error.

Also I would like to mention what this problem is most likely related to Docker because we are also getting the same issue while running strace inside our local environment which runs as a Docker container. I saw some previous SO posts mention adding some argument to "docker run" command to fix it, but not sure is that even possible to do inside ECS task.

Solution

  • From this blog post

    Fargate tasks now support the CAP_SYS_PTRACE Linux capability

    In 2017, we introduced support to add Linux capabilities to ECS Tasks. When we launched AWS Fargate the same year, we decided to disable these options because we wanted to minimize the surface area of attack to offer a secure platform.

    We have since received feedback from customers (as well as partners that gravitate around security and compliance) that they could make good use of some of these capabilities (i.e. CAP_SYS_PTRACE). There are a number of observability tools that can help customers for which compliance is important to achieve the visibility they need. For example, some of these customers have expressed a need to run tools such as strace.

    It is for this reason that, starting with the availability of Fargate platform version 1.4.0, we are allowing customers to enable this specific capability in their Fargate task definitions (across all available Fargate platform versions). Note that CAP_SYS_PTRACE is the only capability that can be added to Fargate tasks at this time. Other capabilities remain supported for tasks running on the EC2 launch type.

    TL/DR: you need to add this parameter to your task definition. And of course making sure you are using Fargate Platform Version 1.4 (or LATEST which maps to 1.4 today).