Search code examples
pythonairflowpython-import

Running a python file that imports from Airflow package, requires airflow instance?

I am running into a weird import issue with Airflow. I want to create a module from which others can import. I also want to run unit tests on this module. However, I noticed that as soon as you import anything from the airflow package, it will try and run Airflow.

Example:

# myfile.py
from airflow import DAG

print("Hello world")

Then run it with python myfile.py, results in:

(.venv) c:\Users\Jarro\Development\airflow-tryout-import>python myfile.py
WARNING:root:OSError while attempting to symlink the latest log directory
Traceback (most recent call last):
  File "c:\Users\Jarro\Development\airflow-tryout-import\myfile.py", line 1, in <module>
    from airflow import DAG
  File "C:\Users\Jarro\Development\airflow-tryout-import\.venv\Lib\site-packages\airflow\__init__.py", line 68, in <module>
    settings.initialize()
  File "C:\Users\Jarro\Development\airflow-tryout-import\.venv\Lib\site-packages\airflow\settings.py", line 559, in initialize
    configure_orm()
  File "C:\Users\Jarro\Development\airflow-tryout-import\.venv\Lib\site-packages\airflow\settings.py", line 237, in configure_orm
    raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: Cannot use relative path: `sqlite:///C:\Users\Jarro/airflow/airflow.db` to connect to sqlite. Please use absolute path such as `sqlite:////tmp/airflow.db`.

Aside from the error itself, I am actually way more concerned that it seems I am not able to import things from Airflow, without there being side-effects (such as database initializations). Am I going all wrong about this? Is there another way I can import things from Airflow without these side effects, for example for typing purposes?

Solution

  • This is happening because of how impoorts work in python. You're importing from the airflow package which has an __init__ file. If you inspect it, there's a piece of code in it that does all of the airflow init stuff.

    lines 67-68 in __init__py:
    if not os.environ.get("_AIRFLOW__AS_LIBRARY", None):
    settings.initialize()

    If you just want to import things for typing like you said, you can set the _AIRFLOW__AS_LIBRARY env variable to anything and it will not initialize.