Airflow DAG - Not able to parse {{ ds }}

Question

I have an Airflow DAG and I use {{ ds }} to get the logical date. As per Airflow documentation template {{ ds }} return logical date in format YYYY-MM-DD in string format. So I am using following code to manipulate the dates

(datetime.strptime('{{ dag_run.logical_date|ds }}', '%Y-%m-%d') - timedelta(3)).strftime('%Y-%m-%d')

but getting following error

Broken DAG: [/usr/local/airflow/dags/custom_dags/feature_store_daily.py] Traceback (most recent call last):
  File "/usr/lib/python3.10/_strptime.py", line 568, in _strptime_datetime
    tt, fraction, gmtoff_fraction = _strptime(data_string, format)
  File "/usr/lib/python3.10/_strptime.py", line 349, in _strptime
    raise ValueError("time data %r does not match format %r" %
ValueError: time data '{{dag_run.logical_date|ds}}' does not match format '%Y-%m-%d'

I am not able to figure out why I am getting this error.

Please advice

Answer 1

Firstly I thought the problem could be de lack of spaces before and after the "|" operand, but I´ve tested and works fine.

In this case, the main problem is that you´re not using the Jinja template where it is effectively parsed. Due to this, the error is that you don´t have a date to format, but a string as it is.

task_xx = SomeOperator(
    ...
    a="{{ dag_run.logical_date|ds }}",
    b="{{dag_run.logical_date | ds}}",
    c="{{ dag_run.logical_date | ds }}"
)

And then, in the SomeOperator these attributes must be in template_fields so it can be parsed:

class SomeOperator(BaseOperator):
    template_fields = ['a', 'b', 'c']

   ....

Now, when you use these attributes, you´ll get what you want. I´ve tested and works fine.

Or, you can also use Jinja templates with fields that already accept this.