Search code examples
pythonkubernetesairflowbitnami

airflow on kubernetes: How to install extra python packages?

i installed airflow using bitnami repo: To install extra python packages i mounted an extra volumeenter image description here

I prepared my requirements.txt file and then i created a ConfigMap using kubectl create -n airflow configmap requirements --from-file=requirements.txt after this i upgraded airflow using helm upgrade....

But in my dags file, i'm still getting the error "ModuleNotFoundError: No module named 'yfinance'"

Solution

  • Posting this as Community wiki for better visibility. Feel free to expand it.

    As @Elpis wrote in the comments section, he followed this guide to install Apache Airflow on Kubernetes.

    And he solved the problem by adding extraVolumeMounts and extraVolumes to the worker pod, and also to the web pod and to the scheduler pod.

    extraVolumeMounts:
  - name: requirements
    mountPath: /bitnami/python/

## Add extra volumes
extraVolumes:
  - name: requirements
    configMap:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
      name: requirements

    After that, all extra Python packages were installed.