I am running my application and metricbeat on K8s. My application has an endpoint that exports prometheus metrics and metricbeat is able to pickup the metrics and add them to elastic.
The problem is that for a given document that makes it into elastic, the field name has the k8 namespace added to it. For example a given document will have a field named prometheus.my-namespace-1.runtime_memory_max_heap.value
This creates a problem since it means that each app that I add to my cluster will force a reindex on Kibana in order to detect the field type. Furthermore it means that I cannot make a general dashboard for my metrics and then filter on namespaces because each app has custom field names.
I would like the field name to just be called prometheus.runtime_memory_max_heap.value
As it turns out the issue was with my configuration. I had the following:
config:
- module: prometheus
period: 15s
metricsets: ["collector"]
enabled: true
hosts: ["${data.host}:9273"]
ssl.verification_mode: "none"
namespace: "${data.kubernetes.namespace}"
Once I updated the namespace it worked. I did try to remove namespace altogether but that did not work so I used a static value.
config:
- module: prometheus
period: 15s
metricsets: ["collector"]
enabled: true
hosts: ["${data.host}:9273"]
ssl.verification_mode: "none"
namespace: "metric"