I am trying to monitor Strimzi using the kube-prometheus-stack helm chart. I have set it up following the tutorial from the official Strimzi documentation. In this tutorial, they both use Podmonitors and a Prometheus config to get some metrics. But I do not quite understand why I need to set up a Podmonitor for some metrics and add jobs in prometheus.prometheusSpec.additionalScrapeConfigs for others. Could someone explain the difference to me?
The PodMonitor(s) are used to select the metrics from the pods from the Strimzi provided custom resources, like for example Kafka, ZooKeeper, KafkaBridge, and so on. The Prometheus operator translates that configuration in corresponding jobs with a kubernetes_sd_configs having role: pod.
The prometheus-additional.yaml file that is used for the additional scrape configs field, contains "raw" jobs configuration for Kubernetes related metrics directly from nodes and provided by cadvisor and kubelet (i.e. volume disk space, CPU and memory usage). In the Prometheus operator there is no a corresponding thing for role: node, doesn't exist a thing like NodeMonitor.
I hope that it makes more sense now.