Airflow connections and short-lived passwords

Question

One of the problem we have when implementing airflow is that most of our connections seem to be a challenge for it.

After somewhat solving connection over SSH, I am now facing the issue of short-lived passwords: a few of our connection requires password to be asked to another service. These passwords always expire after 1h and the only way to get a new one is to query an internal tool using the command line.

Any idea of how I could solve this issue?

Answer 1

You can call your internal tool through the command line to get the password and store it in an Airflow variable. Put this task as the beginning of the DAG where you'll need it. If multiple dags need the password, create a dedicated DAG to refresh the password and run it every 30 minutes for example.

You can store the password in a single task using a PythonOperator.