Search code examples
pythonairflowdockeroperator

How to run a python interactive shell in an docker operator?


What I would like to do is run a docker image in a DockerOperator. While inside the container, I would like to spin up a python interactive shell for a .py file and run a method inside the file.

Is this possible in one operator?


Solution

  • I am not sure what you mean by "interactive shell" - but I assume it is only about simply running a new python interpreter (rather than it being interactive and a user providing some code on-the-flight - that would be rather difficult).

    Yes. You can put your python file next to your DAG and exclude it from parsing as DAG vi .airflowignore file (using regexps) - you can read more here: https://airflow.apache.org/docs/apache-airflow/stable/modules_management.html

    Also just can prepare a custom image with your python file and necessary libraries added.

    Here are the instructions on how to do it with plenty of examples: https://airflow.apache.org/docs/docker-stack/build.html

    Also - and this is the nicest and newest way - if you are using Airlfow 2.2+, you can use @task.docker operator https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#using-the-taskflow-api-with-docker-or-virtual-environments and in this case you can simply decorate a python callable with @task.docker and the python code will be executed using the docker image you specify (using DockerOperator under the hood)