If a container or pod exceeds its memory limit(OOMKilled-137), Does that Pod get rescheduled to another Node or what happened?

Question

Base on my understanding about Kubernetes ,if pod consumes more memory than it mentioned limit. E.g. If mentioned limit is 128Mi,But Pod try to consume memory > 128Mi Then it becomes crashloopback and after that if memory consumption not reduced it becomes "OOMKilled" status. that is handled by Linux kernel. My confusion is after pod become OOMKilled ,does it reschedule in to the same node or different node or its already killed and not recover back?

I tried it with my aks cluster. I saw pod become crashloopback and finally OOMKilled. But after that I could not see any new pod created in the node. Please someone explain what is the next recovery mechanism that Kubernetes use to recover this pod?

      resources:
        requests:
          memory: "64Mi"
          cpu: "0.1"
        limits:
          memory: "128Mi"
          cpu: "0.3"

Answer 1

Pod eviction happens when Node not ready and faulty. and it will be rescheduled to another node which is healthy. That is decided based on concerns like toleration Seconds and default taints values of the Node. OOMKilled will only be reported for containers that have been terminated by the kernel OOM killer. It's important to note that it's the container that exceeds its memory limit that gets terminated (and by default restarted) as opposed to the whole pod (which can very well have other containers). Evictions on the other hand happen at the pod level, and are triggered by Kubernetes (specifically by the Kubelet running on every node) when the node is running low on memory*. Pods that have been evicted will report a status of Failed and a reason of Evicted.

Details about the reason can be seen in the Kubelet events for pod evictions

You need to verify the details run

kubectl describe pod [name]

It should show something like this along with reasons under events.

Coming to your question- Will the pod get reschuduled or stay dead depends on its node.kubernetes.io/not-ready and node.kubernetes.io/unreachable tolerations and RestartPolicy You can verify your restart policy under spec.template.spec.restartPolicy

Reference-

Deployments | Kubernetes

Exceeds memory

Pods reschedule