We are using a database (Apache Geode) that is written in Java. Our servers have 64g of RAM, so we set our Java heap (Xms and Xmx) at about 62g of RAM.
Most Java recommendations I've seen for situations like these is to use the CMS garbage collector, and to set the CMSInitiatingOccupancyFraction at somewhere around 68% (give or take a little, but not much).
But my question is: Why can't we set garbage collection to start at 95% instead of 68%? It seems maybe wasteful to run Java in such a fashion that you can never use more than 68% of your heap without causing non-stop Garbage Collections.
It's bugging us because we are at a stage where our database is doing non-stop garbage collections, and it's hard to justify more RAM when really the JVM has like 18 gigs free. :)
Thanks in advance for any advice.
I think the answers in the comments, plus some things I've managed to figure out, pretty much answers my question.
It seems one of the concerns is fragmentation. You can have 20% of your heap free (80% used), and yet your heap can be badly fragmented to where it's difficult to find the large contiguous blocks necessary for operation. If you start running GC's earlier, like at 70%, you keep fragmentation a little more under control by reclaiming unused objects and creating bigger holes for new objects.
For a non-compacting GC like CMS, there's no 100% guarantee that you'll never become too fragmented to allocate a block of memory, but if you collect early and often you can make that happen as infrequently as possible.