Search code examples
graphtitangremlintinkerpoptinkerpop3

KryoException: Buffer too small

I'm using Gremlin 3.0.2 together with Titan 1.0.0.

The request I send to the Gremlin Server will return a list of nodes and their properties. Effectively, it's a list of items like the following:

[coverurl:[https://lh3.googleusercontent.com/RYb-duneinq8ClWVLVKknkIx1jOKm64LjreziFApEnkKME8j9tHNDRi9NMA6PK4PTXO7], appname:[Slack], pkgid:[com.Slack]]

In one case, a request will return 38 items like the one above and everything is fine. In another case, the list would contain 56 of these items and I get the following exception:

WARN  org.apache.tinkerpop.gremlin.driver.MessageSerializer  - Response [PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 0)] could not be deserialized by org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0.
ERROR org.apache.tinkerpop.gremlin.driver.Handler$GremlinResponseHandler  - Could not process the response
io.netty.handler.codec.DecoderException: org.apache.tinkerpop.gremlin.driver.ser.SerializationException: org.apache.tinkerpop.shaded.kryo.KryoException: Buffer too small: capacity: 0, required: 1
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)

This problem has been discussed here last year. However, for different versions of Titan and for writing data to Titan, instead of reading like it is the case here.

I don't see any programmatic way to adapt the buffer size of the (De)serializer so what is the preferred way to deal with this problem? Also, setting some limit (which?, where?) to some higher value can only be a temporary solution since I never know how much data a request will return.

Anyways - The amount of data I receive is fairly small (probably a little more than 8500 Bytes). I'm surprised that this exception is thrown at all?

Solution

  • Titan 1.0 is based on TinkerPop 3.0.1... are you building Titan on your own?

    TINKERPOP-817 introduces a fix that allows a bufferSize parameter to be configured. As Stephen mentioned in the comments:

    the kryo buffer size was defaulted to 4096 and would those throw that "Buffer too small" exception

    The fix went into TinkerPop 3.0.2 and is documented here.

    In order to use this, you'll need to upgrade your Titan Server to run with TinkerPop 3.0.2, and it would be best to recompile from source after modifying the tinkerpop.version in the Titan pom.xml. Find the Titan build directions here. Alternately, you could consider building the titan11 branch for the latest available fixes and TinkerPop 3.1.1 (Hadoop 2 support!).

    Next, you will need to configure the bufferSize on the appropriate serializer in the gremlin-server.yaml configuration. I do not think you cannot fix this problem with a client configuration only.

    serializers:
  - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { bufferSize: 8192, useMapperFromGraph: graph }}       # application/vnd.gremlin-v1.0+gryo

    Since you have a Java client and you're expecting to work directly with the Vertex objects, perhaps you could consider doing a direct connection to Titan and avoid this serialization completely.