Search code examples
ubuntuairflowairflow-scheduler

Airflow DAG folder is not visible

I have just installed Airflow in ubuntu and I am following tutorials to create DAGs. After successfully installing airflow I have created dags folder in airflow home and created a sample DAG file as sample.py. But I actually don't see the dags folder 

yash@DESKTOP-TUTMG4J:~/airflow$ ls -l
total 316
-rw-rw-rw- 1 yash yash      0 Jun 12 19:26 airflow-webserver.err
-rw-rw-rw- 1 yash yash    140 Jun 12 19:29 airflow-webserver.log
-rw-rw-rw- 1 yash yash    222 Jun 12 19:29 airflow-webserver.out
-rw-r--r-- 1 yash yash      5 Jun 13 14:53 airflow-webserver.pid
-rw-rw-rw- 1 yash yash  36462 Jun 12 17:02 airflow.cfg
-rw-r--r-- 1 yash yash 270336 Jun 13 14:54 airflow.db
drwxrwxrwx 1 yash yash    512 Jun 12 21:57 logs
-rw-rw-rw- 1 yash yash   2578 Jun 12 17:02 unittests.cfg
yash@DESKTOP-TUTMG4J:~/airflow$

But when I start the airflow webserver it says Filling up the DagBag from /home/yash/airflow/dags

yash@DESKTOP-TUTMG4J:~/airflow$ airflow initdb
DB: sqlite:////home/yash/airflow/airflow.db
[2020-06-13 14:51:35,133] {db.py:378} INFO - Creating tables
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
Done.
yash@DESKTOP-TUTMG4J:~/airflow$ airflow webserver
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-06-13 14:53:22,260] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:22,260] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2020-06-13 14:53:23 +0530] [5308] [INFO] Starting gunicorn 19.4.0
[2020-06-13 14:53:23 +0530] [5308] [INFO] Listening at: http://0.0.0.0:8080 (5308)
[2020-06-13 14:53:23 +0530] [5308] [INFO] Using worker: sync
[2020-06-13 14:53:23 +0530] [5313] [INFO] Booting worker with pid: 5313
[2020-06-13 14:53:23 +0530] [5314] [INFO] Booting worker with pid: 5314
[2020-06-13 14:53:23 +0530] [5315] [INFO] Booting worker with pid: 5315
[2020-06-13 14:53:23 +0530] [5327] [INFO] Booting worker with pid: 5327
[2020-06-13 14:53:23,678] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:23,679] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
[2020-06-13 14:53:23,836] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-06-13 14:53:23,837] {dagbag.py:396} INFO - Filling up the DagBag from /home/yash/airflow/dags
[2020-06-13 14:53:23,949] {__init__.py:51} INFO - Using executor SequentialExecutor

and ofcourse I don't see my DAG in airflow UI as well. Can someone enlighten me what I am doing wrong and how can airflow pick my DAGS from the folder.

Solution

  • It looks like you actually didn't create the dags directory in Airflow's home. Airflow's home in your case is /home/yash/airflow/ and there is obviously no dags directory there.

    The Airflow's log message Filling up the DagBag from /home/yash/airflow/dags may be somewhat misleading. It actually means that Airflows is looking for DAG scripts under that path, but it does not mean that the path exists.

    Create the directory:

    mkdir /home/yash/airflow/dags

    and try creating a DAG script again.