Let me start by saying that my understanding of how JNA and Java direct native memory allocations is visceral at best, so I'm trying to describe my understanding of what's going on. Any corrections in addition to responses would be great...
I'm running an application that mixes Java and C native code using JNA and am running accross a reproducible issue with the Java Garbage Collector failing to free references to direct native memory allocations, resulting in the C heap running out of memory.
I'm positive that my C application is not the source of the allocation issue, as I'm passing a java.nio.ByteBuffer into my C code, modifying the buffer, and then accessing the result in my Java function. I have a single malloc and a single corresponding free during each function call, but after repeatedly running the code in Java the malloc will eventually fail.
Here's a somewhat trivialized set of code that exhibits the issue -- realistically I'm trying to allocate about 16-32MB on the C heap during the function call.
My Java code does something like:
public class MyClass{
public void myfunction(){
ByteBuffer foo = ByteBuffer.allocateDirect(1000000);
MyDirectAccessLib.someOp(foo, 1000000);
System.out.println(foo.get(0));
}
}
public MyDirectAccessLib{
static {
Native.register("libsomelibrary");
}
public static native void someOp(ByteBuffer buf, int size);
}
Then my C code might be something like:
#include <stdio.h>
#include <stdlib.h>
void someOp(unsigned char* buf, int size){
unsigned char *foo;
foo = malloc(1000000);
if(!foo){
fprintf(stderr, "Failed to malloc 1000000 bytes of memory\n");
return;
}
free(foo);
buf[0] = 100;
}
Trouble is after calling this function repeatedly the Java heap is somewhat stable (it grows slowly), but the C function eventually cannot allocate any more memory. At a high level I believe this is because Java is allocating memory to the C heap, but not cleaning up the ByteBuffer that points at this memory because the Java ByteBuffer object is relatively small.
Thus far I've found running the GC manually in my function will provide the required cleanup, but this seems like both a poor idea and a poor solution.
How can I manage this problem better so that the ByteBuffer space is appropriately freed and my C heap space is controlled?
Is my understanding of the problem incorrect (is there something I'm running improperly)?
Edit: adjusted buffer sizes to be more reflective of my actual application, I'm allocating for images approximately 3000x2000...
I think that you've diagnosed properly: you never run out of Java heap, so the JVM doesn't garbage collect, and the mapped buffers aren't freed. The fact that you don't have problems when running GC manually seems to confirm this. You could also turn on verbose collection logging as a secondary confirmation.
So what can you do? Well, first thing I'd try is to keep the initial JVM heap size small, using the -Xms command-line argument. This can cause problems, if your program is constantly allocating small amounts memory on the Java heap, as it will run GC more frequently.
I'd also use the pmap tool (or whatever its equivalent is on Windows) to examine the virtual memory map. It's possible that you're fragmenting the C heap, by allocating variable-sized buffers. If that's the case, then you'll see an every larger virtual map, with gaps between "anon" blocks. And the solution there is to allocate constant-size blocks that are larger than you need.