One application I have to deal with regularly launches shell helpers using ProcessBuilder. For reasons untold, it still runs on a 32bit JVM (Sun, 1.6.0.25) even though the underlying OS is 64bits (RHEL 5.x for what it's worth).
This application is memory-happy, so the heap size is set to its maximum of 3 GB, and the permgen is 128M.
However... At random moments, shell helpers fail to launch. Not because of an OutOfMemory, but ENOMEM... The only cause I can see for this is lack of address space.
Well, sure, but at the same moment, the memory is not really under pressure and top reports that the actual memory usage of the JVM and its virtual set size, is not even 3 GB...
Looking at what can be looked of the code of Process, I see that the core method is called forkAndExec(), which is pretty much self explanatory... From what I know of both syscalls, it just shouldn't fail. But it does. And not always.
Why?
edit: it should be noted that neo4j is used. It seems to use FileChannel a lot, can that be the cause of lack of address space?
I would decrease the heap size. The amount of heap actually used could be leaving less and less space for the forked process to run (it inherits resources from its parent)
It is highly likely that just upgrading to a 64-bit JVM would fix the problem, Can you try Java 6 update 30 64-bit instead (just to see if it would fix the problem) If it does or does not, it should tell more about what the cause is (and then you can decide if its worth switching)