Search code examples
airflowscheduled-tasksdirected-acyclic-graphsgoogle-cloud-composer

How can I schedule my DAG to run every 5 minutes?

I am using GCP cloud composer and have an airflow instance and a DAG.

The DAG's default arguments are these:

# Define the default arguments for the DAG.
default_args = {
    # If the start date is set to yesterday, Cloud Composer schedules the workflow to start immediately after the DAG
    # uploads.
    "start_date": datetime.datetime(2021, 12, 2) #yesterday
    ,"owner": "foobar"
    ,"schedule_interval": "*/5 * * * *"
    ,"tags": ["create_empty_dataset", "BigQuery"]

The DAG is running fine, but the schedule (every 5 minutes) is not. I tried using datetime.timedelta(minutes=5) but it didn't work. It just runs once and that's it. That's all I see:

DAG

Any ideas? I can edit and share the DAG code if necessary. It's simple.

Thanks!

Solution

  • Your issue is with how you set the parameters. default_args is set on DAG level but it contains parameters which are passed to operators. So when you set schedule_interval in default_args it means that this parameter is passed to all operators in your DAG and this has no meaning because the operators don't have schedule_interval parameter.

    You should set schedule_interval in the DAG constractor:

    DAG = (..., schedule_interval="*/5 * * * *", default_args=default_args)