Whenever I tried to run my DAG it gets success state but it can't able to run the task. While running the first time it works perfectly
trigger_controller_dag.py:
def conditionally_trigger(context, dag_run_obj):
c_p = context['params']['condition_param']
if context['params']['condition_param']:
dag_run_obj.payload = {'message': context['params']['message']}
pp.pprint(dag_run_obj.payload)
return dag_run_obj
dag = DAG(
dag_id='example_trigger_controller_dag',
default_args={
"owner": "airflow",
"start_date": datetime.utcnow(),
},
schedule_interval='@once',
)
trigger = TriggerDagRunOperator(
task_id='test_trigger_dagrun',
trigger_dag_id="example_trigger_target_dag",
python_callable=conditionally_trigger,
params={'condition_param': True, 'message': 'Hello World'},
dag=dag,
)
trigger_target_dag.py:
args = {
'start_date': datetime.utcnow(),
'owner': 'airflow',
}
dag = DAG(
dag_id='example_trigger_target_dag',
default_args=args,
schedule_interval=None,
)
def run_this_func(ds, **kwargs):
print("Remotely received value of {} for key=message".
format(kwargs['dag_run'].conf['message']))
run_this = PythonOperator(
task_id='run_this',
provide_context=True,
python_callable=run_this_func,
dag=dag,
)
While running the DAG I am getting the following error, dependency 'Task Instance State' FAILED: Task is in the 'success' state which is not a valid state for execution. The task must be cleared in order to be run
Your start_date
variables in both DAGS are dynamic and may be causing your problem 'start_date': datetime.utcnow()
. Setting them to a dynamic start is not recommended and leads to errors.
Try setting it to a static start date like 'start_date': datetime(2019, 5, 29) #year month day
Taken from Airflow FAQ
We recommend against using dynamic values as start_date, especially datetime.now() as it can be quite confusing. The task is triggered once the period closes, and in theory an @hourly DAG would never get to an hour after now as now() moves along.
Another SO question on this: why dynamic start causes issues