How does Prometheus know when a pod crashed?

Question

I understand with Prometheus we can set up alerting rules which can detect and raise an alert if a pod crashes.

I want to understand how does Prometheus itself know when a pod crashed or is stuck in pending state.

• Does it know this when it is trying to scrape metrics from pod's http endpoint port?

OR

• Does Prometheus get the pod status information from Kubernetes?

The reason why I'm asking this is because I want to set up Prometheus to monitor existing pods that I have already deployed. I want to be alerted if a pod keeps crashing or if it is stuck in pending state. And I want to know if Prometheus can detect these alerts without making any modifications to the code inside the existing pods.

Answer 1

The common way for prometheus to extract metrics and health is by the use of scraping (thru an http endpoint is the most common). Since pods can have multiple containers, it is best to scrape an http endpoint of your running container.

If prometheus didnt receive a good response from this endpoint, it can determine that the container is down.

Prometheus itself does not do alerting, you normally delegate that to the alert manager.