Hey guys I faced some issues with exposing metrics to /federate endpoint by Prometheus
I see that for all metrics it drops # HELP lines from each metric. Only # TYPE stays. Tried to search on documentation and forums can't find any boolean answer like "Prometheus Federation by default drops # HELP line and leaves only # TYPE and the metric"
Someone may be faced it in the past and have some knowledge to share?
Configuration is not really affecting here AFAIK since even scraping K8s endpoints with just Kubernetes-pods job without any drops/replacements but providing federate="yes" and then in federation endpoint, there are no # HELP lines, they just dropped by default.
In actual target for pods scraped by Prometheus to /metrics or /monitoring endpoint # HELP is persist
The Prometheus /federate endpoint is intended to be consumed by other Prometheus instances and so it exposes only what's strictly needed by another Prometheus instance. While the # TYPE line is essential because it defines the type of the metric, the # HELP line apparently isn't regarded as essential because there's not done much with it inside Prometheus.
See for example here:
The HELP text in Prometheus’ exposition format was invented with the intent of creating functionality later.
Currently, Prometheus provides a metadata API that allows to query the # HELP text of a metric. But that's about all Prometheus does with the # HELP text internally, it's not interpreted in any other ways.
So, in summary, Prometheus currently doesn't expose the # HELP line on its federation endpoint because it regards it as non-essential.
However, this might change in the future as more functionality is added. If you have a specific use case, you can always raise a feature request in this area.