Trying to run same airflow task across different DAGs

Question

So, I have the same tasks running across different DAGs, since the DAGs run at different schedules, they overlap which is something I don't want.

I looked into the various cross-dag dependency techniques mentioned here: https://docs.astronomer.io/learn/cross-dag-dependencies?tab=traditional#externaltasksensor but nothing really fits my use case since for database option I won't be able to add normal scheduling and for the others I need to explicitly decide the dependency flow

I essentially want max_runs for a specific task across DAGs to be 1, I don't care about the order. The solution I came up with is to set airflow variable as task name as locked when running a task and on being done mark it as unlocked and make the other tasks wait till it's locked.

Is there a better way to do this?

Answer 1

The correct approach would be to use a task pool with capacity 1. This will allow only one task assigned to this pool to be executing at given time.

your_task = YourOperator(
    task_id="your_task",
    pool="your_pool_name",  # << applicable to any operator
    dag=dag,
    # other task params...
)