I have a java tomcat application deployed on open shift environment and I have observed pod memory is not releasing even after the load has stopped (No Traffic). I have waited till 1 day but still, memory not released due to which pods are never scaling down. In HPA properties also, utilized memory is above 80%.
I have already analyzed the heap dump of application and there is no memory leak at all, also never observed any out of memory error/exception.
Tried using below JVM props which I found on open shift blog (given below) but still the problem is not resolved. https://cloud.redhat.com/blog/scaling-java-containers
About Application
It's bit of traditional application (Advance Java & Servlets) built in monolithic architecture and using tomcat 8 as a application server.
It'd be really helpful if anyone can help or share their expertise in scaling java application on open shift.
This is the way that the Java JVM works: it never releases memory. And, assuming a modern JVM, it will generally just allocate as much memory as available in the container on startup. There are a bunch of reasons for this, but the gist of it is that the JVM is doing its own memory management and memory garbage collection so it needs to have as big of a pool of memory available as it can.
As such, memory usually isn't the best option for autoscaling Java applications. Typically you will want to autoscale based on CPU and just set the amount of memory for each pod based on load testing (and generally set memory request to the same amount as memory limit). e.g. If I have a CPU request size of 2000 mCPU, I'll do performance benchmarking on a 2000 mCPU container and gradually reduce the available memory until I notice a degradation in performance or unacceptable GC. That then establishes my pod resources and I can use HPA to scale up and down based on CPU.
Standard disclaimers apply: performance tuning is a very application specific thing. But autoscaling Java based applications on memory usage isn't going to work effectively.