Global variables in Airflow

Question

I am trying to implement basic ETL job, using Airflow, but stucked in one point:

I have 3 functions. And I want to define global variables for each of them like:

function a():
   return a_result

function b():
     use a
     return b_result
function c():
     use a and b

And then use these functions in python_callable.

Defining as usual global a_result is not working. Any solutions?

Answer 1

As I wrote in my comment,

When you return something in your python_callable, you can access the returned value if you pass the task context to the next operator. https://airflow.apache.org/concepts.html?highlight=xcom

The following is semi-pseudo code that illustrates the idea

# inside a PythonOperator called 'pushing_task' 
def push_function(): 
    return value 

# inside another PythonOperator where provide_context=True 
def pull_function(**context): 
    value = context['task_instance'].xcom_pull(task_ids='pushing_task')

pushing_task = PythonOperator('pushing_task', 
                              push_function, ...)

pulling_task = PythonOperator('pulling_task', 
                              pull_function, 
                              provide_context=True ...)