Here is my situation: I have a task at hand that requires lots of memory. I do not have enough ram, and no matter what I tried (Jrockit with /3gb switch etc), I can't give JVM enough ram and the operation is terminated with an exception, telling me I need more heap space.
Is there any way I can force the JVM to use the OS's swapping mechanism so that it won't run out of memory? This is Windows xp 32 bit
It would take ages, but I would not care, I just need this operation to be completed.
I've run out of options, and I have no control over any of the variables here..
This is a required edit, since I am having the same response from pretty much everyone :) This is not my code. Someone has written a tool that reads an xml file into a repository. The tool uses EMF, and loads the whole model at once. All I can do is to feed it the XML file. In case of native code running under Windows or Linux etc, the OS provides memory to it, using virtual memory/swap space, and the app does not know about it. I was wondering if it is possible to do the same with the JVM. Under Windows 32 bit, -Xmx can go up to a certain amount, but that is not enough. Going out and buying new hardware is not an option for me for the moment. So I was wondering if it is possible to make the JVM work like native processes. Slow, but still working. Apparently that is not possible, and I am out of luck. I just need to know if I'm really out of options.
Apparently there is one way around the limits of Java heap. It is even used in a commercial product called BigMemory which basically allows you to have almost unlimited memory by transparently swapping out to OS swap and/or to disk if needed.
The idea is to use direct ByteBuffers to store your objects data. Because direct byte buffers' contents are stored in native process memory (as opposed to heap) you can rely on OS swap mechanism to swap memory out for you. I found this on this website (search for 'direct byte buffer' on the page).
Here is how you can implement it (java-pseudo-code'ish):
class NativeMemoryCache{
private Map<Object, ByteBuffer> data = new HashMap<...>();
public void put(Object key, Serializable object){
byte[] bytes = serialize(object);
//allocate native memory to store our object
ByteBuffer buf = ByteBuffer.allocateDirect(bytes.length);
buf.put(bytes);
buf.flip();
data.put(key, buf);
}
public Object get(Object key){
ByteBuffer buf = data.get(key).duplicate();
byte[] bytes = new byte[buf.remaining()];
buf.get(bytes);
return deserialize(bytes);
}
private byte[] serialize(Object obj){ ... }
private Object deserialize(byte[] bytes){ ... }
}
Hope you get the idea. You just need to implement the serialization (you can also compress your objects using zip. This will be effective if you have few big objects especially ones containing zippable data like strings).
Of course NativeMemoryCache object, data hash map and keys will be in heap, but that should not take much memory.