The AWS credentials are not accessible under a non-root user when using the Apache Airflow image

Question

Here's my dockerfile and part of docker-compose file with slight modification of original apache airflow compose file. Dockerfile

FROM apache/airflow:2.7.2
ENV PIP_USER=false

RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}"


RUN python3 -m venv /opt/airflow/project1-env
COPY mobile-sa-esd/requirements.txt .
RUN /opt/airflow/project1-env/bin/pip install -r requirements.txt


RUN python3 -m venv /opt/airflow/project2-env
COPY mobile-sa-jobs/requirements.txt .
RUN  /opt/airflow/project2-env/bin/pip install -r requirements.txt

ENV PYTHONPATH="$PYTHONPATH:/opt/airflow/project1-env/bin:/opt/airflow/project2-env/bin"

#ENV PIP_USER=true

USER root

RUN apt-get update && \
    apt-get install -y curl unzip groff

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \
    unzip awscliv2.zip && \
    sudo ./aws/install

docker-compose.yml

    volumes:
    - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
    - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
    - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
    - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
    - $HOME/Desktop/project1:/opt/airflow/project1
    - $HOME/Desktop/project2:/opt/airflow/project2
    - $HOME/.aws:/root/.aws:ro
  user: "${AIRFLOW_UID:-50000}:0"

There is an issue where my application can't access AWS credentials under the non-root user. I suspect this is because the Airflow image uses the non-root user "airflow." I've set "USER root" in the Dockerfile for two reasons: to install utilities and to have access to AWS credentials inside the container.

I can verify that AWS credentials are accessible by logging into the Airflow container as root and checking the profiles.

docker exec  -u root  -it <container_id> bash

aws configure list-profiles
profile_1
profile_2

However, when logged in as the non-root user, the AWS CLI doesn't recognize the profiles, indicating an issue with access to AWS credentials.

docker exec  -it df82e4191398 bash
aws configure list-profiles
default@df82e4191398:/opt/airflow$

As you can see response is empty. I acknowledge the possibility of achieving this by setting environment variables in the Docker container. However, I've come across information suggesting that this approach may not be advisable for production, so I've chosen not to pursue it.

I attempted to mount the AWS credentials directory under $HOME/.aws:/.aws:ro, but this change did not resolve the issue.

Answer 1

You have this in your volumes:

volumes:
...
- $HOME/.aws:/root/.aws:ro

So the credentials get mapped to /root which is not accessible for your non-root user. You need to map it properly:

volumes:
...
- $HOME/.aws:/home/user/.aws:ro

Where user is the name of your user. If you don't have a home folder for your airflow user, you should create one. Or give the new user the privileges to access /root which might not be very secure. You might need to write a script, which creates a home folder probably.