Set PYTHONPATH in MWAA

Question

I'm trying to use local module inside a dag on MWAA.

The folder structure looks like :

.
├── __init__.py
├── dags
│   ├── __init__.py
│   └── my_dag
│      ├── __init__.py
│      └── dag.py
│   └── utils
│      ├── __init__.py
│      └── file.py
│      └── secrets.py
│      └── date.py

I try to use functions from ./dags/utils/secrets by importing them like :

from dags.utils.secrets import get_secret

Locally, I've been able to make it works by setting environment variable PYTHONPATH to usr/local/airflow

Is it the best way ? If not how can I make it works on MWAA ?

Thank you,

Answer 1

Locally, I've been able to make it works by setting environment variable PYTHONPATH to usr/local/airflow

Is it the best way ? If not how can I make it works on MWAA ?

When deploying Airflow to an MWAA environment, you don't explicitly set the PYTHONPATH environment variable.

I try to use functions from ./dags/utils/secrets by importing them like :

from dags.utils.secrets import get_secret

Adjust the Python import statement relative to the MWAA environment's DAGs folder. For example, if the DAGs folder is s3://<bucket>/dags, then the import statement would be:

from utils.secrets import get_secret

Example DAGs folder:

s3://<bucket>/dags/__init__.py
s3://<bucket>/dags/my_dag/__init__.py
s3://<bucket>/dags/my_dag/dag.py
s3://<bucket>/dags/utils/__init__.py
s3://<bucket>/dags/utils/file.py
s3://<bucket>/dags/utils/secrets.py
s3://<bucket>/dags/utils/date.py