Search code examples
jsonscalajacksonpermgenjerkson

Jerkson Json parser for scala.

I have used Jerkson for scala, to serialize my list of objects to a JSON file. I'm able to decompose the object into JSON format object and written to a file. Now, I when I want to read it into my program for further processing I get this error. FYI, my file size is 500MB and in the future might grow upto 1GB.

I saw few forums which has asked to increase the XX:MaxPermSize=256M. I'm not sure if this is going to solve my problem, even if it does for now, what is the guarantee that this might not surface later when the size of my JSON file grows to 1GB.Is there a better alternative ? Thanks!

Exception in thread "main" java.lang.OutOfMemoryError: PermGen space
    at java.lang.String.intern(Native Method)
    at org.codehaus.jackson.util.InternCache.intern(InternCache.java:41)
    at org.codehaus.jackson.sym.CharsToNameCanonicalizer.findSymbol(CharsToNameCanonicalizer.java:506)
    at org.codehaus.jackson.impl.ReaderBasedParser._parseFieldName(ReaderBasedParser.java:997)
    at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:418)
    at com.codahale.jerkson.deser.ImmutableMapDeserializer.deserialize(ImmutableMapDeserializer.scala:32)
    at com.codahale.jerkson.deser.ImmutableMapDeserializer.deserialize(ImmutableMapDeserializer.scala:11)
    at org.codehaus.jackson.map.ObjectMapper._readValue(ObjectMapper.java:2704)
    at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1315)
    at com.codahale.jerkson.Parser$class.parse(Parser.scala:83)
    at com.codahale.jerkson.Json$.parse(Json.scala:6)
    at com.codahale.jerkson.Parser$class.parse(Parser.scala:14)
    at com.codahale.jerkson.Json$.parse(Json.scala:6)
Solution

  • From the stack trace we can see that Jackson interns the Strings that are parsed as the names of fields in your document. When a String is interned, it is put in the PermGen, which is the part of the heap that you are running out of. I reckon this is because your document has many, many different field names - perhaps generating with some naming scheme? Whatever the case, increasing you MaxPermSize might help some, or at least delay the problem, but it won't solve it completely.

    Disabling String interning in Jackson, on the other hand, should solve it completely. The Jackson FAQ has more information about what configuration options to tweak: http://wiki.fasterxml.com/JacksonFAQ#Problems_with_String_intern.28.29ing