Search code examples
airflowdbt

No such file or directory - Airflow

I have my airflow project with the structure as below

airflow
    |
    |----dags
    |      |----dag.py
    |
    |----dbt-redshift
             |----models
                   |----model.sql

I have included the dbt-redshift directory in the volumes section as

volumes:
  -./dbt-redshift:/opt/airflow/dbt-redshift

And I'm trying to run the dbt inside the DAG using a bash operator

 dbt_task = BashOperator(task_id='dbt', bash_command="cd ~/dbt-redshift && dbt run", dag=dag)

But when i execute the DAG i get the error

cd: /home/***/dbt-redshift no such file or directory

I'm not sure I understand how these directories are located inside the airflow project.

Solution

  • You are mounting the volume inside the container to /opt/airflow/dbt-redshift, but the BashOperator references ~/dbt-redshit with ~ resolving to /home/airflow.

    (Assuming you are using the apache/airflow image)

    Either change the command used by the BashOperator to reference /opt/airflow/dbt-redshift or change the volume to mount to the home directory.