Airflow: Failing to push data back into the pipeline

Question

I have setup a workflow, which consists of two tasks:

In second, I am successfully pulling data pushed into the pipeline by the first task. However, after processing data, when I try to push data back into the pipeline, I get the error "INFO - Task exited with return code -9".

Here are the logs:

Why is XCOM failing to push data into pipeline? How can I communicate data to other tasks?

Answer 1

Identified issue:

The problem was with the cross communication i.e. XCOM, which is not suitable for passing dataframes, and large amounts of data in the pipelines. Reference: https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/xcoms.html

Solution:

Since we can not pass the data between tasks using XCOMs, we can use shared storage using which tasks can communicate data. The XCOMs can be used to pass the location of the files in shared storage so that other tasks can know from where to fetch data from.