Why does containerized Spring Application allocates too much memory in Kubernetes Cluster?
I have an issue that some of our microservices have a memory allocation spike that triggers an OOMKiller Kubernetes event.
The issue doesn't happen that often but enought to be noticed. The service is not doing anything crazy at the moment when it's happening, just normal load. Also, interesting fact that memory doesn't gradually increase, on the contrary - it's like a sudden explosion.
I also did some deep analysis on what apps are doing at the time of mem explosion - all seem normal to be normal which is why I don't think it's some problem or memory leak in the code. Plus same behaviour happens across different services that are not connected with each other in almost any way. Also, i most of the cases after restarting service proceeding with the work where it was left off. Which is why I'm leaning towards the idea that it's some kind of Framework or Software problem.
More on the setup:
- Spring Boot (2.4.x - 2.5.x). Different stacks(Web and WebFlux) and dependencies (Spring Cloud, Spring Kafka, ElasticSearch, Postgres, etc) are used accross microservices.
- Java 11 (latest OpenJDK image)
- The apps are running on one of the newest versions of Kubernetes cluster(AKS).
- Before we had no resource limits implemented for microservices. So in case of an incident, we had an infinite memory allocation until it eats all of the memory of K8S node which forces everything to get down & restart. Currently we have resource limits for pods implemented to avoid that behaviour.
Questions:
- I'm wondering if it's a JVM/Spring issue / memory leak. Maybe someone had a similar experience?
- Are there any tools / approaches you can recommend to take a snapshot of the pod before it's getting killed to have an ability to analyze that later?
We were able to catch metrics at the time when it's happening. Turned out, that memory limits were misconfigured in helm charts and default values were taken by JVM. This leaded to situtation when JVM had a lot (around 3GB) memory allocated but only used some (150MB). So our K8s node was experiencing some memory starvation at the time when it was running lots of in the mode like that mode.
And yes, Instana turned out to be quite helpful tool to debug that. See metrics listed below:
- Keep numbers which appear in both columns, in J lang
- Differentiation in J
- How to reshape an array with an arbitrary size in one dimension?
- Why is Insert (fold) right associative
- Write 4 : 'x&{.&.;: y' tacitly
- Alignment issue when printing formatted prime numbers in J language
- How can I define a verb in J that applies a different verb alternately to each atom in a list?
- How to get user input in the J programming language
- How to unbox a list of boxed lists of differing lengths in J?
- How can I fix 'noun result was required' error in J?
- In j, how can I define a verb locally in one scope and pass it to a defined adverb?
- Convert boxed array to normal array?
- Read column of CSV file as array
- Replace atom in array of strings
- What does the dyad `=` do to boxed strings?
- Index of minimum element using J
- How can I take the outer product of string vectors in J?
- Building an array of verbs in J
- Reading in multidigit command line parameter
- Amend with bond to new data shows unexpected behaviour
- How to turn a table or matrix into a (flat) list in J
- How to run dissect in J?
- How to define selection using index function in J
- How to exit the J console?
- Find 4-neighbors using J
- Writing custom verbs in J
- How do I negate a selector in J lang?
- How to use arbitrary selector in interchange in J lang?
- different result once square root is added inside tacit
- Sum of arrays with repeated indices