I am looking for a way to filter out airflow metrics for other dags send over through StatsD. There is this document in airflow showing how to setup statsD which is responsible for metric gathering in airflow. It is possible to filter for metric prefixes but this technically means it will send all those prefixes to statsd which will pose a great cost if it is happening for all the dags.
[metrics]
statsd_allow_list = scheduler,executor,dagrun
Since I am using datadog it is very important to reduce cost by minimizing the metrics needed to send over to datadog.
Is there anyway to limit the metrics for a specific dag?
I find the answer to this, It is not possible to filter based off a dag name or a specific dag. But we can include the complete metric name including the dag name in the filter list.
for example here is the metric name: dag.example_dag.some_task.duration
all need to do is to add this in the airflow.cfg like below:
[metrics]
statsd_allow_list = dag.example_dag.some_task.duration, <other filters>