In my app start up, I'm creating trove long hash set with 75*10^6 capacity.
Profiler shows, that app use 1.4g for it. I'm try to set -Xmx1600m, and catch out of memory.
-Xmx2000m same.
-Xmx2030m same.
-Xmx2040m OK.
Profiler doesn't see so big allocation, where and why it's happen?
Most likely your tenured space is not large enough. The jvm might not be smart enough to shrink other regions to allow you such a large continuous block. Note CMS doesn't defrag so you can get this problem with much smaller arrays.
For a collection that big you might consider using off heap memory.