Search code examples
airflowsmtplib

Airflow email_on_failure not finding SMTP connection information

I have Airflow 2.2.4 running in AWS EKS, deployed with the official Helm chart (version 1.6.0). I am trying to set up email notifications upon DAG failure. I have this section in my values file:

config:
  smtp:
    smtp_host: "smtp.gmail.com"
    smtp_starttls: "True"
    smtp_ssl: "False"
    smtp_mail_from: "my.address@gmail.com"
    smtp_port: 587

I have also configured Airflow to use AWS Secrets Manager as the backend, so secrets in Secrets Manager prefixed with, e.g., airflow/connections become Airflow connections.

I have created an AWS Secrets Manager secret named airflow/connections/smtp_default and containing:

{
  "login": "my.address@gmail.com",
  "password": "blahblahblah"
}

I can confirm that Airflow is aware of this connection by execing into the scheduler pod and running airflow connections get smtp_default.

The secret is structured the way it is because the source code suggests that's how it should be done. Yet, I see this:

[2023-06-10, 16:10:58 UTC] {configuration.py:381} WARNING - section/key [smtp/smtp_user] not found in config
[2023-06-10, 16:10:58 UTC] {email.py:214} INFO - Email alerting: attempt 1
[2023-06-10, 16:11:08 UTC] {email.py:214} INFO - Email alerting: attempt 2
[2023-06-10, 16:11:18 UTC] {email.py:214} INFO - Email alerting: attempt 3
[2023-06-10, 16:11:28 UTC] {email.py:214} INFO - Email alerting: attempt 4
[2023-06-10, 16:11:38 UTC] {email.py:214} INFO - Email alerting: attempt 5
[2023-06-10, 16:11:48 UTC] {configuration.py:381} WARNING - section/key [smtp/smtp_user] not found in config
...
...

Am I misreading the source code? Is there something else that could be the cause of this that I can't think of?

Edit: It doesn't appear that Airflow is correctly parsing the AWS Secrets Manager secret:

>>> from airflow.hooks.base import BaseHook
>>> conn = BaseHook.get_connection("smtp_default")
>>> conn.login is None
True
>>> conn.password is None
True
>>> # Somehow it's shoving part of the secret into the single field conn.schema
>>> conn.schema
'  "login": "my.address@gmail.com",  "password": "blahblahblah"}'

But this seems to contradict the documentation for the Secrets Manager backend; namely, the example they provide:

As an example, if you have set connections_prefix as airflow/connections, then for a connection id of smtp_default, you would want to store your connection at airflow/connections/smtp_default. This can be done through the AWS web console or through Amazon CLI as shown below:

aws secretsmanager put-secret-value \
    --secret-id airflow/connections/smtp_default \
    --secret-string '{"login": "nice_user", "password": "this_is_the_password", "host": "ec2.8399.com", "port": "999"}'
Solution

  • This is a bug in the Amazon provider. I don't know which release did it, but upgrading to Airflow 2.6.0 fixed the issue.