Search code examples
prometheuspromql

Why does `count_over_time` increase without matching raw data?

... or "How does count_over_time work?"

Given the following vector and value:

kube_job_created{
  app="prometheus",
  chart="prometheus-7.1.0",
  component="kube-state-metrics",
  heritage="Tiller",
  instance="100.96.2.23:8080",
  job="kubernetes-service-endpoints",
  job_name="foobar-24150",
  kubernetes_name="prometheus-kube-state-metrics",
  kubernetes_namespace="devops",
  namespace="devops",
  nodename="ip-1-1-1-1.ap-southeast-2.compute.internal",
  release="prometheus"
} 1538008393

Count over the entire vector returns 1:

count(kube_job_created{app="prometheus",chart="prometheus-7.1.0",component="kube-state-metrics",heritage="Tiller",instance="100.96.2.23:8080",job="kubernetes-service-endpoints",job_name="braze-user-attribute-sync-24150",kubernetes_name="prometheus-kube-state-metrics",kubernetes_namespace="devops",namespace="devops",nodename="ip-10-10-4-112.ap-southeast-2.compute.internal",release="prometheus"})

Result:

{} 1

Now, with count_over_time it's different depending on the range vector selector:

1m:

count_over_time(kube_job_created{app="prometheus",chart="prometheus-7.1.0",component="kube-state-metrics",heritage="Tiller",instance="100.96.2.23:8080",job="kubernetes-service-endpoints",job_name="braze-user-attribute-sync-24150",kubernetes_name="prometheus-kube-state-metrics",kubernetes_namespace="devops",namespace="devops",nodename="ip-10-10-4-112.ap-southeast-2.compute.internal",release="prometheus"}[1m])

Result is "2":

{app="prometheus",chart="prometheus-7.1.0",component="kube-state-metrics",heritage="Tiller",instance="100.96.2.23:8080",job="kubernetes-service-endpoints",job_name="braze-user-attribute-sync-24150",kubernetes_name="prometheus-kube-state-metrics",kubernetes_namespace="devops",namespace="devops",nodename="ip-10-10-4-112.ap-southeast-2.compute.internal",release="prometheus"} 2
  • 1h returns 120
  • 24h returns 2880

What exactly is it counting over time?

Solution

  • count_over_time counts the number of samples in the range, for each time series. Like all _over_time functions it works across all the samples of one time series at a time.

    If you wanted the number of time series in an instant vector, then the count aggregator will tell you that.