Search code examples
dockeremailsmtpairflow

airflow smtp not working with docker - section/key [smtp/smtp_user] not found in config / OSError: [Errno 99] Cannot assign requested address

I am trying to send a simple email from my airflow using the local docker setup.

My dag looks like this:

default_args = {
        "owner": "Airflow",
        "start_date": pendulum.today('UTC').add(days=-1),
        "depends_on_past": False,
        "email_on_failure": True,
        "email_on_retry": False,
        "email": "[email protected]",
        "retries": 1,
        "retry_delay": timedelta(minutes=5)
    }
with DAG(dag_id="send_emails", schedule="@daily", default_args=default_args, catchup=False) as dag:

    sending_email_notification = EmailOperator(
        task_id="sending_email",
        to="[email protected]",
        subject=f"New email",
        html_content=f"""
            <h3>New Data is available</h3>
        """
        )

    sending_email_notification

I have setup the [smtp] section in airflow.cfg which is in the same folder as my docker-compose.yaml as such:

[smtp]
smtp_host = smtp-mail.outlook.com
smtp_starttls = True
smtp_ssl = False
smtp_user = [email protected]
smtp_password = mypassword
smtp_port = 587
smtp_mail_from = [email protected]

I have also tried doing this from gmail using an app password

[smtp]
smtp_host = smtp.gmail.com
smtp_starttls = True
smtp_ssl = False
smtp_user = [email protected]
smtp_password = appssecret
smtp_port = 587
smtp_mail_from = [email protected]
smtp_timeout = 30
smtp_retry_limit = 5

However the DAG fails everytime with the following error and traceback:

[2024-01-31, 21:24:42 UTC] {taskinstance.py:1400} INFO - Marking task as FAILED. dag_id=send_emails, task_id=sending_email, execution_date=20240131T211339, start_date=20240131T212442, end_date=20240131T212442
[2024-01-31, 21:24:42 UTC] {configuration.py:1050} WARNING - section/key [smtp/smtp_user] not found in config
[2024-01-31, 21:24:42 UTC] {email.py:270} INFO - Email alerting: attempt 1
[2024-01-31, 21:24:42 UTC] {configuration.py:1050} WARNING - section/key 
[smtp/smtp_user] not found in config
[2024-01-31, 21:24:42 UTC] {email.py:270} INFO - Email alerting: attempt 1
[2024-01-31, 21:24:42 UTC] {taskinstance.py:2007} ERROR - Failed to send email to: [email protected]


During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2005, in handle_failure
self.email_alert(error, task)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2422, in email_alert
send_email(task.email, subject, html_content_err)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 80, in send_email
 return backend(
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 154, in send_email_smtp
send_mime_email(e_from=mail_from, e_to=recipients, mime_msg=msg, conn_id=conn_id, dryrun=dryrun)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 272, in send_mime_email
smtp_conn = _get_smtp_connection(smtp_host, smtp_port, smtp_timeout, smtp_ssl)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/email.py", line 316, in _get_smtp_connection
return smtplib.SMTP(host=host, port=port, timeout=timeout)
  File "/usr/local/lib/python3.8/smtplib.py", line 255, in __init__
 (code, msg) = self.connect(host, port)
  File "/usr/local/lib/python3.8/smtplib.py", line 339, in connect
self.sock = self._get_socket(host, port, self.timeout)
  File "/usr/local/lib/python3.8/smtplib.py", line 310, in _get_socket
return socket.create_connection((host, port), timeout,
  File "/usr/local/lib/python3.8/socket.py", line 808, in create_connection
raise err
  File "/usr/local/lib/python3.8/socket.py", line 796, in create_connection
sock.connect(sa)
OSError: [Errno 99] Cannot assign requested address
[2024-01-31, 21:24:42 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 105 for task sending_email ([Errno 99] Cannot assign requested address; 557)
[2024-01-31, 21:24:42 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 1
[2024-01-31, 21:24:42 UTC] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check

The part that strikes me is WARNING - section/key [smtp/smtp_user] not found in config but I'm a bit lost as to what is going on here. If someone could both help fix this but also explain why this is happening that would be for the future.

Solution
    1. To rule out credential or user access issues can you try to send to a test SMTP server such as this https://www.wpoven.com/tools/free-smtp-server-for-testing A few months back on docker Airflow (version 2.5.2 default Celery Executor) was able to test Email operator by following these steps: Adding these env variables to specify a SMTP server to the docker compose file.

    AIRFLOW_SMTP_SMTP_HOST: 'smtp.freesmtpservers.com' AIRFLOW_SMTP_SMTP_PORT: '25' AIRFLOW_SMTP_SMTP_STARTTLS: 'False'

    Dag task code for email:

    task_send_email = EmailOperator(
    task_id='task_send_email',
    to='[email protected]', 
    subject='test email', 
    html_content='Testing email',
)
    1. Once the above step works, you can try to test with either outlook or gmail test. In my case, I had IT create a service email account (piggybacked to my LDAP but had it's own password) to test this out. I did not set the SMTP_USER and SMTP_PASSWORD in airflow config. Also note that in my case the outlook SMTP host is different than yours (just in case). I also added the service account credentials in the Airflow connection_id named 'smtp_default'

    AIRFLOW__SMTP__SMTP_HOST: 'smtp.office365.com' AIRFLOW__SMTP__SMTP_PORT: '587' AIRFLOW__SMTP__SMTP_STARTTLS: 'True' AIRFLOW__SMTP__SMTP_SSL: 'False' AIRFLOW__SMTP__SMTP_MAIL_FROM: '[email protected]'

    task_send_email = EmailOperator(
    task_id='task_send_email',
    conn_id='smtp_default',
    to=['[email protected]','[email protected]'],
    cc='[email protected]',
    subject='testing airflow email using email connection type test', 
    html_content='Testing email from airflow DAG',
    do_xcom_push=True,
    on_success_callback =success,
    on_failure_callback =failure
)

    Not sure about the OSError: [Errno 99] there are some other stackoverflow threads that may help you out.