Search code examples
kubernetesmonitoringprometheusgrafanapromql

How to get CPU and memory usage of pod in percentage using promethus

I want to display pod details in the following format using promql/Prometheus.

Image1

Furthermore, I want to display CPU and memory utilization of application/component in below format using promql

Image2

promql query: sum(container_memory_working_set_bytes) by (pod)

I can get the consumed memory by pod using above query.

How to calculate percentage of memory used ? I am not able to fetch memory limit of stateful pod using promql Could you please suggest any query/API details ?

Solution

  • Per-pod CPU usage in percentage (the query doesn't return CPU usage for pods without CPU limits)

    100 * max(
  rate(container_cpu_usage_seconds_total[5m])
    / on (container, pod)
  kube_pod_container_resource_limits{resource="cpu"}
) by (pod)

    The kube_pod_container_resource_limits metric can be scraped incorrectly if scrape config for kube-state-metrics pod is improperly configured. In this case the original pod label for this metric is moved to the exported_pod label because of honor_labels behavior - see these docs for details. In this case label_replace function must be used for moving exported_pod label to pod label:

    100 * max(
  rate(container_cpu_usage_seconds_total[5m])
    / on (container, pod)
  label_replace(kube_pod_container_resource_limits{resource="cpu"}, "pod", "$1", "exported_pod", "(.+)")
) by (pod)

    Per-pod memory usage in percentage (the query doesn't return memory usage for pods without memory limits)

    100 * max(
  container_memory_working_set_bytes
    / on (container, pod)
  kube_pod_container_resource_limits{resource="memory"}
) by (pod)

    If the kube_pod_container_resource_limits metric is scraped incorrectly as mentioned above, then the label_replace function must be used for moving exported_pod label value to pod:

    100 * max(
  container_memory_working_set_bytes
    / on (container, pod)
  label_replace(kube_pod_container_resource_limits{resource="memory"}, "pod", "$1", "exported_pod", "(.+)")
) by (pod)