I have installed and configured Prometheus on my Azure Kubernetes as per the following configuration
prometheus_values.yaml
======================
global:
scrape_interval: 1m
scrape_timeout: 10s
evaluation_interval: 1m
# Forcing Kubelet metrics scraping on http
kubelet:
enabled: true
serviceMonitor:
https: false
# Disabling scraping of Master Nodes Components
kubeControllerManager:
enabled: false
kubeScheduler:
enabled: false
kubeEtcd:
enabled: false
kubeProxy:
enabled: false
scrape_configs:
- job_name: dev-go-app
static_configs:
- targets: ['40.x.x.x:8080']
- job_name: default-go-app
static_configs:
- targets: ['40.x.x.x:8080']
and installed prometheus as shown below
Installing prometheus
=====================
helm upgrade --install prometheus \
--namespace monitoring \
stable/prometheus \
--values prometheus_values.yaml
It looks like I have to manually update the Prometheus configuration and apply for each application which is a difficult task especially when I have 100+ applications in the same Azure Kubernetes cluster.
Is there a way to let Prometheus automatically discovery the Kubernetes services and start pulling the metrics?
Instead of using bare Prometheus, I suggest setting up a kube-prometheus-stack (formerly prometheus-operator): https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
This stack has a built-in discovery mechanism in it - a CRD called ServiceMonitor: https://coreos.com/operators/prometheus/docs/latest/user-guides/running-exporters.html. This is a selector which will auto-discover your services thru labels, further discover pods behind these services, and load them into Prometheus scrape config.