I've been unable to find an answer to this after days of Googling. I have a service running in Marathon/Mesos. I have a Prometheus cluster scraping metrics. My Marathon metrics port config looks like this:
{
"containerPort": 8081,
"hostPort": 0,
"servicePort": 31301,
"protocol": "tcp",
"labels": {
"metrics": "/metrics"
}
}
Prometheus, configured with just a boilerplate marathon-sd config, successfully finds this target, but it then listens for metrics on: __address__ = [NodeIP]:31301; so it's listening on the host's IP with the service port, rather than the dynamically assigned host port, while the service port only matters for Marathon-LB.
I know that Marathon defines the environment variable $PORT0 in the container to be the host port, however I can't figure out how to access this from the Prometheus SD config, nor how to access other fields dynamically configured by Marathon like Endpoints. Does anyone have any suggestions? I can't/don't want to assign a static hostPort because I have more containers than there are physical nodes in the cluster, as is sort of the point with container orchestration.
It is a known bug in Prometheus: it uses the servicePort Marathon app definition property instead of the hostPort one. It is fixed in the v2.6.0.