Search code examples
kubernetesout-of-memorycgroups

Kubernetes pod memory vs cgroup

I am running apache in Kubernetes pod and I am troubled by random OOM kills of the apache process and I am unsure of the cause.

The pod has 600M memory limit and it seems it can consume it quite easily even though PHP has 128 MB limit per run. When observing dmesg -w I can regularly see:

[4019392.626796] Memory cgroup out of memory: Kill process 4178127 (apache2) score 1137 or sacrifice child
[4019392.636520] Killed process 4178127 (apache2) total-vm:143960kB, anon-rss:22856kB, file-rss:10472kB, shmem-rss:28228kB
  1. How can I be sure, that this OOM kill is related to the machine or to the pod limits?
  2. There are several apache pods and I am not sure how to recognize the pod to which this process belonged. Is there a way?
  3. It would be great if these were accessible within the pods, so one could see killed threads which actually belonged to the pod (ideally in error output). Is it possible?
Solution

  • Covering your questions in the same order:

    1. kubectl get events in the namespace where you run a pod. You will see Kubernetes OOM events among others. If there's nothing - it will be the kernel itself.

    2. There's a project called pid2pod, but it greps only what's still running. The other option will be to route dmesg to stdout (on container OS level) and implement centralized log collection with Kubernetes metadata enrichment. Then every log event will have pod id in the metadata.

    3. Not sure what are you trying to solve here. Usually, one pod (and I assume one container) is one process. Once it killed by OOM, there's nothing else that can take an action on this inside the pod. It's the controller's (Deployment) responsibility to react to this event and recreate the pod.

      If you just want to watch for OOMs within the remaining pods you may implement smth like kubewatch. Unfortunately, I don't have a project that lets you subscribe within a pod directly, but this can be used as an example. Or in case of centralized logging, you can subscribe to a log stream inside a pod.