How to send AWS MWAA cluster metrics to DataDog?
AWS's Managed Workflows for Apache Airflow (MWAA) provides cluster metrics like the CPUUtilization and MemoryUtilization for the BaseWorker, Scheduler and WebServer. I would like to have these in DataDog so I can set usage alerts. Here's a example of what they look like in Cloudwatch:
However, DataDog's official MWAA integration doesn't have these metrics listed. Is there a way to directly specify the metrics and forward them to DataDog? Or, possibly, are they already being picked up but under an obscure namespace?
After some research I found that you need to activate an option to send custom metrics from AWS.
You have to follow the next steps:
Datadog UI → Integrations → Search for AWS → Select your account → Go to Metric Collection → Turn on Collect custom metrics from Cloudwatch
After some minutes search for something like aws.mwaa.cpuutilization and it should be there for you to filter by cluster or any other type of query
- How do we trigger single airflow dag multipler times using TriggerDagRunOperator?
- How to consider daylight savings time when using cron schedule in Airflow
- Airflow EMRServerlessCreateApplicationOpertor can't detect application name from airflow input parameter using jinja template
- Airflow Task Group Execution Order
- How to Configure Airflow Celery executor with redis sentinel?
- Running airflow tasks/dags in parallel
- Facing Issue in passing metadata field with DataprocCreateClusterOperator (Airflow 2.0)
- Airflow Helm Chart Worker Node Error - CrashLoopBackOff
- Airflow: how to delete a DAG?
- How to update airflow.cfg file in MWAA local Runner
- Airflow Scheduler Hanging When Run with Remote Database
- Connection refused when calling Postgres container from another docker container
- How to run Airflow Web Console on different port?
- Service account cannot get resource \"sparkapplications/status\" in API group \"sparkoperator.k8s.io\"
- Is there a way to pass the dataproc version via the Airflow DataprocCreateBatchOperator method?
- What is the difference between min_file_process_interval and dag_dir_list_interval in Apache Airflow 1.9.0?
- Apache Airflow unit and integration test
- Airflow it's running two jobs and returning error in Databricks but ran succesfully
- How to use AirFlow to run a folder of python files?
- Unexpected String Concatenation Issue in Airflow 2.7.2 DAG
- Not uploading file to postgres with bulk_load. Apache Airflow
- The task completes with the SKIPPED status if it was started outside the interval of the schedule_interval parameter
- Datadog exclude the logs
- Capture airflow run duration
- Airflow DAG throws GlueJobOperator is not JSON serializable
- Ariflow Tasks in TaskGroup run out of order
- Python doesn't see added module to the same dir
- Use SlackNotifier in Airflow within a callback function
- AWS Airflow v2.0.2 doesn't show Postgres connection type
- Airflow kills my tasks after 1 minute