Search code examples
pythonairflowairflow-scheduler

Reading Airflow rest API passed conf in script

Hello I am trying to use the experimental Airflow rest API. Below is my Dag definition:

from airflow import DAG
from airflow.operators import BashOperator,PythonOperator
from datetime import datetime, timedelta

seven_days_ago = datetime.combine(datetime.today() - timedelta(7),
                                  datetime.min.time())

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': seven_days_ago,
    'email': ['[email protected]'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
}

dag = DAG('python_test', default_args=default_args)
t1 = BashOperator(
    task_id='testairflow',
    bash_command='python ${AIRFLOW_HOME}/dags/python_test.py',
    dag=dag)

and below is my script which simply reads the passed parameters and prints it.


import argparse


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Argument... ')
    parser.add_argument('--env', metavar='path', required=True, default='dev', help='Execution Environment')
    parser.add_argument('--config_path', required=True, help='Custom config s3 path..')
    parser.add_argument('--data_lake_bucket', required=False, default="s3://dl_bucket")
    args = parser.parse_args()

    print("--------------------------------------")

    print(args.env)
    print(args.config_path)
    print(args.data_lake_bucket)


    print("--------------------------------------")

And below is my curl command to trigger it.

curl -X POST \
  http://localhost:8080/api/experimental/dags/python_test/dag_runs \
  -H 'Cache-Control: no-cache' \
  -H 'Content-Type: application/json' \
  -d '{"conf":"{\"--env\":\"dev\", \"--config_path\":\"this_is_conf\"}"}'

I am trying to figure out how to read the passed parameters with -d in the python script or any other job? Current I see the job is getting triggered but doesn't read the parameters.

Solution

  • Experimental API has been deprecated in Airflow 2 and will be removed in Airflow 3. I heartily recommend switching to Airflow 2 and switching to a fully-fledged REST API which has been implemented as part of the Airflow 2 effort and provides quite comprehensive, well structured, documented REST API with all the bells and whistles you can imagine. It's build with OpenAPI specification, it has Swagger UI, authentication support, we even have generated Python client to use the API: https://github.com/apache/airflow-client-python.

    Also it is "Stable" and follows Airflow 2 SemVer versioning, so as long as you use Airflow 2, you have backwards-compatibility guarantees.

    https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html

    Airflow 1.10 (if you are still using it) has reached end-of-life on June 17 and it will not receive any more fixes (even critical fixes). In fact there have already been security fixes in Airflow 2 that has not been (and will not be) back-ported to 1.10, so it's highest time to upgrade.

    You can also watch Airflow Summit's discussion panel about why you SHOULD upgrade https://airflowsummit.org/sessions/2021/panel-airflow-security/