Search code examples
prometheusgrafanapromql

Why function increase returns unexpected result?


I have promql query

process_resident_memory_bytes{job="integrations/kubernetes/kube-apiserver", cluster=~"my-cluster"}

Which gives following data in grafana portal

Timestamp Value
2023-09-23 17:30:00 1097273344
2023-09-23 17:30:30 1097273344
2023-09-23 17:31:00 1098895360
2023-09-23 17:31:30 1098895360
2023-09-23 17:32:00 1082576896
2023-09-23 17:32:30 1082576896
2023-09-23 17:33:00 1116635136
2023-09-23 17:33:30 1116635136
2023-09-23 17:34:00 1110745088
2023-09-23 17:34:30 1110745088
2023-09-23 17:35:00 1072701440
2023-09-23 17:35:30 1072701440
2023-09-23 17:36:00 1056124928
2023-09-23 17:36:30 1056124928
2023-09-23 17:37:00 1193168896
2023-09-23 17:37:30 1193168896
2023-09-23 17:38:00 1028886528
2023-09-23 17:38:30 1028886528
2023-09-23 17:39:00 1080430592
2023-09-23 17:39:30 1080430592
2023-09-23 17:40:00 1087283200
2023-09-23 17:40:30 1087283200
2023-09-23 17:41:00 1064062976
2023-09-23 17:41:30 1064062976
2023-09-23 17:42:00 1176289280
2023-09-23 17:42:30 1176289280
2023-09-23 17:43:00 1094549504
2023-09-23 17:43:30 1094549504
2023-09-23 17:44:00 1059459072
2023-09-23 17:44:30 1059459072
2023-09-23 17:45:00 1064747008
2023-09-23 17:45:30 1064747008
2023-09-23 17:46:00 1052381184
2023-09-23 17:46:30 1052381184
2023-09-23 17:47:00 1140494336
2023-09-23 17:47:30 1140494336
2023-09-23 17:48:00 1073520640
2023-09-23 17:48:30 1073520640
2023-09-23 17:49:00 1080832000
2023-09-23 17:49:30 1080832000
2023-09-23 17:50:00 1082146816
2023-09-23 17:50:30 1082146816
2023-09-23 17:51:00 1087004672
2023-09-23 17:51:30 1087004672
2023-09-23 17:52:00 1176756224
2023-09-23 17:52:30 1176756224
2023-09-23 17:53:00 1098498048
2023-09-23 17:53:30 1098498048
2023-09-23 17:54:00 1103314944
2023-09-23 17:54:30 1103314944
2023-09-23 17:55:00 1083441152
2023-09-23 17:55:30 1083441152
2023-09-23 17:56:00 1081024512
2023-09-23 17:56:30 1081024512
2023-09-23 17:57:00 1118924800
2023-09-23 17:57:30 1118924800
2023-09-23 17:58:00 1038856192
2023-09-23 17:58:30 1038856192
2023-09-23 17:59:00 1082642432
2023-09-23 17:59:30 1082642432
2023-09-23 18:00:00 1083723776

After applying the following query, I have obtained the following data.

increase(process_resident_memory_bytes{job="integrations/kubernetes/kube-apiserver", cluster=~"my-cluster"}[10m])
Timestamp Value
2023-09-23 17:50:00 4993515520
2023-09-23 17:50:30 4993515520
2023-09-23 17:51:00 4874217244.444445
2023-09-23 17:51:30 4874217244.444445
2023-09-23 17:52:00 3757775075.555556
2023-09-23 17:52:30 3757775075.555556
2023-09-23 17:53:00 3801151715.555556
2023-09-23 17:53:30 3801151715.555556
2023-09-23 17:54:00 3800628337.777778
2023-09-23 17:54:30 3800628337.777778
2023-09-23 17:55:00 3835139413.3333335
2023-09-23 17:55:30 3835139413.3333335
2023-09-23 17:56:00 4938374257.777778
2023-09-23 17:56:30 4938374257.777778
2023-09-23 17:57:00 3787684977.777778
2023-09-23 17:57:30 3787684977.777778
2023-09-23 17:58:00 4933845902.222222
2023-09-23 17:58:30 4933845902.222222
2023-09-23 17:59:00 4981036373.333334
2023-09-23 17:59:30 4981036373.333334
2023-09-23 18:00:00 4976840248.888889

My understanding increase function formula is value[currentvalue] - value[current-10min].

Example to calculate the increase value at particular timestamp:2023-09-23 18:00:00

1083723776 - 1082146816(i.e value at 2023-09-23 17:50:00) = 1576960

The output I obtained in the second table differs at "2023-09-23 18:00:00." Please guide me on what might be missing when I try to calculate the increase function, or correct me if my understanding is incorrect.

Thanks in Advance.


Solution

  • No, increase is more complex then a simple subtraction. It automatically accounts for counter resets, that are present in data, and thus should only be used for counters.

    Your metric being called process_resident_memory_bytes is not a counter. If you want to simply get change in value over last 10 minutes use either delta function, or if you are concerned about non-integer results something like

    process_resident_memory_bytes{job="integrations/kubernetes/kube-apiserver", cluster=~"my-cluster"} - process_resident_memory_bytes offset 10m