how to import custom modules in Cloud Composer

Question

I created a local project with apache Airflow and i want to run it in cloud composer. My project contains custom modules and a main file that calls them.
Example : from src.kuzzle import KuzzleQuery

Structure:

• main.py
• src
  • kuzzle.py

I have imported my project folder in data storage and when i refreshed the UI of airflow composer i've got this error:

  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/airflow/gcs/dags/quality-control/main.py", line 8, in <module>
    from src.kuzzle import KuzzleQuery, Tag
ModuleNotFoundError: No module named 'src' ```

Answer 1

GCP Composer is loading the dags folder as a python module, so you can just put your code there in a separate file or folder and it will work as usual referencing it from the dags.

dags/
  dag1.py
  util.py

Then you can reference util and import it from dag1 as:

from util import custom_function

If dags folder does not work for you composer also loads more directories like the ones for loading modules.

You can also check this question since it is also related.

Airflow dag dependencies not available to dags when running Google's Cloud Compose