Search code examples
airflowdirected-acyclic-graphs

Check print statement value (not defined under any tasks) in airflow logs


I have used print statement to print the taskgroup. Where can I possibly see the value getting printed? In Airflow we can only see specific task logs I assume. Can we see the print statement value which is not defined under any tasks?

import datetime

from airflow import DAG
from airflow.operators.empty import EmptyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.operators.latest_only import LatestOnlyOperator
from airflow.decorators import dag, task, task_group

def my_function():
    print("hello")

with DAG(
    dag_id="my_dag_name",
    start_date=datetime.datetime(2023, 7, 7),
    schedule="@daily",
    catchup=False
):

    latest_only = LatestOnlyOperator(task_id="latest_only")

    first = EmptyOperator(task_id="first")
    last = EmptyOperator(task_id="last")

    @task_group()
    def group():
        options = ["branch_a", "branch_b", "branch_c", "branch_d"]
        for option in options:
            t = EmptyOperator(task_id=option)
    
    group1 = group()
    group2 = group()
    print(group1)
            
    first >> [group1, group2] >> last
    latest_only >> first

Solution

  • You can see those logs under $AIRFLOW_HOME/logs/scheduler/$date/$dag_name.py.log

    $AIRFLOW_HOME - you airflow home directory

    $date - for example 2023-08-12

    $dag_name.py.log - path of the dag under dags_folder. for example if you create sub_folder by name "temp" and have there my_dag_name.py then it will be temp/my_dag_name.py.log

    enter image description here