Search code examples
javamapdb

Out of memory w/ mapdb


I'm getting an out of memory error when testing mapdb. Given that the whole idea of the project is to serialize data structures to disk and avoid memory problems, I figure I'm doing something wrong. Any ideas what I'm doing wrong? Or is there a bug?

    @Test
    public void testLarge() throws Exception {
        final HTreeMap<UUID, String> storage = DBMaker.newTempHashMap();

        String string = createDataSize(250);

        ArrayList<UUID> keys = new ArrayList<>();
        for (int i = 0; i < 320000; i++) {
            final UUID key = UUID.randomUUID();
            storage.put(key, string);
            keys.add(key);
        }

        for (UUID key : keys) {
            assertNotNull(storage.get(key));
        }

        for (UUID key : keys) {
            storage.remove(key);
        }

        assertEquals("nothing left", 0, storage.size());
    }

    /**
     * Creates a message of size @msgSize in KB.
     */
    private static String createDataSize(int msgSize) {
        // Java chars are 2 bytes
        msgSize = msgSize / 2;
        msgSize = msgSize * 1024;
        StringBuilder sb = new StringBuilder(msgSize);
        for (int i = 0; i < msgSize; i++) {
            sb.append('a');
        }
        return sb.toString();
    }
}

Stack trace. Line 29 in my function corresponds to the "assertNotNull(storage.get(key));" line.

java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Arrays.java:2367)
    at java.lang.String.<init>(String.java:168)
    at org.mapdb.SerializerBase.deserializeString(SerializerBase.java:724)
    at org.mapdb.SerializerBase.deserialize(SerializerBase.java:932)
    at org.mapdb.SerializerBase.deserialize(SerializerBase.java:731)
    at org.mapdb.HTreeMap$1.deserialize(HTreeMap.java:134)
    at org.mapdb.HTreeMap$1.deserialize(HTreeMap.java:123)
    at org.mapdb.StorageDirect.recordGet2(StorageDirect.java:536)
    at org.mapdb.StorageDirect.get(StorageDirect.java:201)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:50)
    at org.mapdb.AsyncWriteEngine.get(AsyncWriteEngine.java:163)
    at org.mapdb.EngineWrapper.get(EngineWrapper.java:50)
    at org.mapdb.CacheHashTable.get(CacheHashTable.java:85)
    at org.mapdb.HTreeMap.get(HTreeMap.java:387)
    at com.sample.StorageTest.testLarge(StorageTest.java:29)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:292)

Based on unholysampler's suggestion, I made the following change which fixed the problem.

diff -r 35918e46551a src/test/java/com/sample/StorageTest.java
--- a/src/test/java/com/sample/StorageTest.java Thu Mar 21 13:40:16 2013 -0600
+++ b/src/test/java/com/sample/StorageTest.java Thu Mar 21 13:42:24 2013 -0600
@@ -16,7 +16,9 @@

     @Test
     public void testLarge() throws Exception {
-        final HTreeMap<UUID, String> storage = DBMaker.newTempHashMap();
+        File tmpFile = File.createTempFile("largeTest", null);
+        DB db = DBMaker.newFileDB(tmpFile).deleteFilesAfterClose().journalDisable().make();
+        final HTreeMap<UUID, String> storage = db.getHashMap("name");

         String string = createDataSize(250);

@@ -25,6 +27,9 @@
             final UUID key = UUID.randomUUID();
             storage.put(key, string);
             keys.add(key);
+            if (i%100==0) {
+                db.commit();
+            }
         }

         for (UUID key : keys) {

Solution

  • The documentation shows that there is a commit() method.

    db.commit();  //persist changes into disk
    

    testLarge() never calls commit(), so nothing has been persisted yet.

    You also aren't calling close(), so the map may still be in memory when you move to another test.

    NOTE: I've never worked with the library, but this is my best guess.