I'm using Apache Airflow to schedule ETL jobs which are python scripts. When I create dags on airflow, it sets dags status on off. My code are like this.
import airflow
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
default_args = {
'owner': 'oguz',
'depends_on_past': False,
'start_date': datetime(2018, 1, 25),
'email': ['airflow@airflow.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
'end_date': datetime(9999, 1, 1),
}
dag = DAG('%s', default_args=default_args, schedule_interval='@daily')
# t1 and t2 are examples of tasks created by instantiating operators
t1 = BashOperator(
task_id='%s',
bash_command='python /bookmark/ETL/extract/incremental/%s.py',
dag=dag)
t2 = BashOperator(
task_id='%s',
bash_command='python /bookmark/ETL/load/incremental/%s.py',
retries=3,
dag=dag)
t2.set_upstream(t1)
I've searched on airflow documentation but I couldn't find anything.
How can I run automatically airflow dags?
Thanks,
If you toggle the status to "ON" in the UI, it should start running your DAG based on the given interval (daily in your case). If you want new DAGs to be enabled by default, you can update set dags_are_paused_at_creation = False
under [core]
in your airflow config.