Search code examples
pythonpython-3.xairflowdirected-acyclic-graphs

DAG not visible in Web-UI

I am new to Airflow. I am following a tutorial and written following code.

from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
from models.correctness_prediction import CorrectnessPrediction

default_args = {
    'owner': 'abc',
    'depends_on_past': False,
    'start_date': datetime.now(),
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

def correctness_prediction(arg):
    CorrectnessPrediction.train()

dag = DAG('daily_processing', default_args=default_args)

task_1 = PythonOperator(
    task_id='print_the_context',
    provide_context=True,
    python_callable=correctness_prediction,
    dag=dag)

On running the script, it doesn't show any errors but when I check for dags in Web-UI it doesn't show under Menu->DAGs

enter image description here

But I can see the scheduled job under Menu->Browse->Jobs

enter image description here

I also cannot see anything in $AIRFLOW_HOME/dags. Is it supposed to be like this only? Can someone explain why?

Solution

  • The ScheduleJob that you see on the jobs page is an entry for the Scheduler. Thats not the dag being scheduled.

    Its weird that your $AIRFLOW_HOME/dags is empty. All dags must live within the $AIRFLOW_HOME/dags directory (specifically in the dags directory configured in your airflow.cfg file). Looks like you are not storing the actual dag in the right directory (the dags directory).

    Alternatively, sometimes you also need to restart the webserver for the dag to show up (though that doesn't seem to be the issue here).