Our data pipeline setup currently makes use of Docker containers running on EC2 instances and Apache Airflow as the orchestration tool. We're trying to move to MWAA for the advantages associated with a managed service(scalability, maintenance, scheduler monitoring etc...). My question is what's the best way to integrate Docker with MWAA? Currently we run our services/app via the DockerOperator on Airflow. Would we need to get Docker running on the MWAA environment? Any suggestions would be appreciated.
I would suggest to go ahead with ECS or ECS Fargate and run your Docker containers there.
MWAA supports open-source integrations with Fargate and other services, so can be easily integrated with Airflow to orchestrate