Apache Airflow - AWS MFA Authentication

Question

I've been running Airflow using helm chart. The object of airflow is to invoke AWS Batch Job in the DAGs like below.

batch = AWSBatchOperator(
    task_id='batch',
    job_name='my-job',
    job_definition='arn:aws:batch:my-job-def',
    job_queue='arn:aws:batch:my-job-queue',
    overrides={
        'command': ["echo", "hello"]
    },
    array_properties={},
    parameters={},
    region_name='my-region',
    dag=dag
)

When I invoke this command, task fails with the log below.

[2020-12-01 08:30:11,642] {awsbatch_operator.py:150} INFO - AWS Batch Job has failed executed
[2020-12-01 08:30:11,660] {taskinstance.py:1150} ERROR - Unable to locate credentials

This seems reasonable because our AWS policy forces users to login with MFA to do something in the AWS.

How can I define Login these information(such as aws_access_key, secret, etc) onto above environemnt? Should it be defined in helm chart's values.yaml or AirFlow UI?

Answer 1

in case you are running airflow on AWS, you should be able to attach an IAM role to the instance (EC2), tasks (ECS) or pod (EKS) so that credentials are not taken into account but rather the IAM role attached.

Also, AWS has a managed airflow service: https://aws.amazon.com/blogs/aws/introducing-amazon-managed-workflows-for-apache-airflow-mwaa/