Why can't I connect to my s3 bucket from an ecs task?

Question

I have an mlflow tracking server with an s3 backend store deployed as per the terraform in this example: https://github.com/Glovo/terraform-aws-mlflow. The terraform has a public and private option, the public one allows you to access the mlflow tracking server and s3 from your local machine. I have tested saving metrics and models to mlflow from my computer, and haver verified that it works. I have set up the aws cli and have authenticated. When I try to run my code in ecs, I get the following error:

botocore.exceptions.NoCredentialsError: Unable to locate credentials

This happens when the code tries to log an artifact in mlflow. I've verified that the ecs execution role has full access to s3, so why does it need credentials, and which credentials does it need?

Any help would be much appreciated.

Answer 1

I've verified that the ecs execution role has full access to s3

The ECS Execution Role is used by the ECS service to run your task. It needs access to things like ECR and Secrets Manager in order to create and run your ECS task. The ECS Execution Role is not provided to the actual code running inside your task.

You need to assign a Task Role to your ECS task. The Task Role is provided to code running inside your ECS tasks to allow that code to access AWS resources.