My system has 3 servers per Node, we have 02 Nodes, and set HeapSize 16Gb for each JVM -> It meant totally we have 06 JVMs to serve service online.
We have 02 layer: layer web published via web facing for customers, 1 layer is application layer to response and reply query for request from web customer.
I monitored day by day, and set notification when RAM used of JVM reach to 90% it will alarm message. and then it auto reduce, My understanding is GC will collect garbage java and free ram when it reach high.
I use ManageEngine Application Management to monitor, and it sent alarm messages on some JVMs. I monitor on ManageEngine Application Management (abbre: AM), AM stills send alarm message JVM reach 90% RAM in both High workloads time (daily working hours: 14:00PM - 15:00 PM) and Low workloads time ( 01:00 AM - 02:00 AM).
My questions: are those situations normal ? should I increase and provide more RAM physicals for these JVMs, i'm worry about OOM, and want system runs stable? Could you explain about GC process ?
Here is an article describing how Java Garbage Collection (GC) works. GC basics are pretty much the same across different Java versions, there are some items in this article that are specific to IBM Java or IBM hardware.
https://www.eclipse.org/openj9/docs/gc/
The JVM requests memory from the OS for various operational needs of the JVM. Typically the biggest portion of JVM memory is used by the Java heap - but the JVM also needs memory that is not part of the heap, commonly referred to as off-heap or native memory.
The JVM automatically manages the memory inside the Java heap. This activity includes allocating memory on the heap when new objects are created, and clearing that memory when the objects are no longer in use. GC is the activity that clears unused memory inside the Java heap.
But clearing memory inside the heap by GC does not return that memory to the OS - rather it just marks the memory as 'free' inside the JVM, so that new objects can be placed in that memory.
So execution of GC in the JVM does not reduce the OS view of RAM used by the JVM.
It is possible for the JVM to return unused memory to the OS, but that is not the typical default behavior. Returning unused Java heap memory to the OS is often not helpful for performance, because the JVM is likely to need to reclaim that memory from the OS when the Java heap grows again.
When deploying Java systems, it is crucial to ensure that there is sufficient RAM for both the Java heap max size AND the off-heap/native elements of the Java runtime. The amount of native heap varies widely based on the type of workload run on the system.
My usual rule of thumb is that the JVM may use 25% more RAM than max heap size, but this number can be smaller or larger by quite a bit depending on the workload. So the only way to know for sure how much RAM the JVM will need is to run the workload and monitor the process size from the OS.
You should monitor the process size of each JVM over time, under load - after some period of running, the java process size should roughly stabilize at some amount of RAM which is more than the Java heap size.
The total RAM used by all the JVMs and all the other processes running on the box should be less than total RAM in the system by some margin, because unusual/unexpected events may cause some process to consume more RAM than usual. If RAM exhaust occurs, the OS may react in different ways - swapping to disk, killing processes using large amounts of RAM, etc. - which are all very bad for your applications' performance.
The 'free RAM margin' that you choose depends on your personal and organizational risk tolerance. Personally, I do not like to see used RAM get above 90% in regular operation.