Search code examples
kubernetesheapstercadvisor

cAdvisor custom metrics with heapster format

I currently have a Kubernetes cluster configured with Heapster/InfluxDB/Grafana.

I know that the Kubelet now has an embedded cAdvisor instance that I have configured to look at an application endpoint to gather custom metrics.

I followed this guide: http://guoshimin.github.io/2016/06/03/custom-metrics.html:

{
  "endpoint" : "http://localhost:31234/sessions",
  "metrics_config" : [
    {
      "name" : "activeSessions",
      "metric_type" : "gauge",
      "units" : "number of active sessions",
      "data_type" : "int",
      "polling_frequency" : 10,
      "regex" : "Active sessions: ([0-9]+)"
    }
  ]
}

Currently the Kubelet is throwing the error -

failed to create collector for container "/docker/664af6c4c1998514770371267ba6c117c532a448f6301f14fc53ca9798abff5b", config "prometheus": json: cannot unmarshal object into Go value of type string

It seems to think that I am using a Prometheus config. Is there a different format I should use in order for it to collect metrics and pass them to Heapster since I do not have Prometheus in my setup?

Solution

  • Kubernetes currently only supports gathering custom metrics in the Prometheus format. Your configuration is for a generic collector, so the prometheus collector isn't able to parse it.

    As an experiment, you could manually change the docker label on the container running on the host from io.cadvisor.metric.prometheus to io.cadvisor.metric.raw, and it should work. There is no mechanism in Kubernetes to do this automatically though.

    Once cAdvisor has scraped the custom metrics, they are then exported in a format the Heapster understands. As long as cAdvisor is able to get the metrics, Heapster will be able to get the metrics from cAdvisor.