Java OutOfMemoryError using Javolution Struct

Question

I'm using Javolution Struct to represent a C struct in Java.

However, every time I create a new item, it is allocated in the native part of the process, and increases the memory usage.

Eventually, the process reaches 4GB of memory, even though the Java heap size remains small. The process is then killed by the OS (I am forced to use a 32-bit java).

Here is a small class demonstrating the issue:

import javolution.io.Struct;

public class StructTest
{
    public static class SimpleStruct extends Struct
    {
        private final Unsigned32 value = new Unsigned32();
    }

    public static void main(String[] args)
    {
        try
        {
            for (int i = 0; i < 10000000 ; i++)
            {
                SimpleStruct st = new SimpleStruct();
                st.value.set(0xFFFF);

                if (i % 10000 == 0)
                {
                    long free = Runtime.getRuntime().freeMemory();
                    long total = Runtime.getRuntime().totalMemory();
                    System.out.printf("%08d: Total Memory=%,12d ; Used Memory=%,12d\n", i, total, total - free);
                }
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }
}

Here is the "top" for this process. As you can see, the memory increases very rapidly:

886 root 20 0 1617m 447m 5536 S 26.8 1.2 0:00.83 java -cp . StructTest
886 root 20 0 1917m 761m 5536 S 28.2 2.1 0:01.12 java -cp . StructTest
886 root 20 0 2116m 990m 5540 S 359.9 2.7 0:04.80 java -cp . StructTest
886 root 20 0 2120m 1.0g 5580 S 115.3 2.8 0:06.00 java -cp . StructTest
886 root 20 0 2302m 1.2g 5580 S 23.5 3.3 0:06.24 java -cp . StructTest
886 root 20 0 2568m 1.4g 5580 S 180.3 4.1 0:08.08 java -cp . StructTest
886 root 20 0 2817m 1.7g 5580 S 95.5 4.8 0:09.09 java -cp . StructTest
886 root 20 0 3114m 2.0g 5580 S 26.4 5.6 0:09.36 java -cp . StructTest
886 root 20 0 3406m 2.3g 5580 S 30.2 6.4 0:09.67 java -cp . StructTest
886 root 20 0 3699m 2.6g 5580 S 25.5 7.3 0:09.93 java -cp . StructTest
886 root 20 0 3994m 2.9g 5580 S 27.4 8.1 0:10.21 java -cp . StructTest

I could try to reuse the struct, instead of recreating it, but I need many items in multiple threads.

Is there a simple way to indicate to the process to free the memory of a struct I no longer need?

Edit: Tested on RedHat Linux (mostly 6.2, but also happens on 5.6). 2.6.32-220.el6.x86_64 Red Hat Enterprise Linux Server release 6.2 (Santiago) Happens with Java 1.6.0_03 (1.6.0_03-b05) and a 64-bit version of 1.6.0_30.

Thanks, Al

Answer 1

Following the post from yoavain, it seems that this is indeed a DirectByteBuffer issue.

After reading the psot "how to garbage collect a direct buffer java", it would seem that normal GC can't handle this ("The contents of direct buffers may reside outside of the normal garbage-collected heap, and so their impact upon the memory footprint of an application might not be obvious").

One solution is to actively call System.gc() from time to time. This seems to clear the memory, but is unreliable and cumbersome.

A better solution appears to use the buffer's cleaner and clean it when we're done with, essentially "free"ing it as c-code.

Therefore, the revised test class would be:

import javolution.io.Struct;
import sun.nio.ch.DirectBuffer;

import java.nio.ByteBuffer;

public class StructTest
{
    public static class SimpleStruct extends Struct
    {
        private final Unsigned32 value = new Unsigned32();
    }

    public static void main(String[] args)
    {
        try
        {
            for (int i = 0; i < 10000000 ; i++)
            {
                SimpleStruct st = new SimpleStruct();
                st.value.set(0xFFFF);

                if (i % 10000 == 0)
                {
                    long free = Runtime.getRuntime().freeMemory();
                    long total = Runtime.getRuntime().totalMemory();
                    System.out.printf("%08d: Total Memory=%,12d ; Used Memory=%,12d\n", i, total, total - free);
                }
                ByteBuffer byteBuffer = st.getByteBuffer();
                if (byteBuffer instanceof DirectBuffer)
                {
                    ((DirectBuffer)byteBuffer).cleaner().clean();
                }
            }
        }
        catch (Exception e)
        {
            e.printStackTrace();
        }
    }
}

When I ran this, the memory seemed to remain steady.