Search code examples
javajvmapache-flinkflink-streaming

Java 11 + Docker Based App = `java.lang.OutOfMemoryError: Direct buffer memory`

With a Flink 1.14.3 application running on Kubernetes using the flink:1.14.3-scala_2.12-java11 Docker image, I'm getting this Java memory error:

Caused by: java.lang.OutOfMemoryError: Direct buffer memory
    at java.base/java.nio.Bits.reserveMemory(Unknown Source)
    at java.base/java.nio.DirectByteBuffer.<init>(Unknown Source)
    at java.base/java.nio.ByteBuffer.allocateDirect(Unknown Source)
    at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:648)
    at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:623)
    at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:202)
    at io.netty.buffer.PoolArena.tcacheAllocateSmall(PoolArena.java:172)
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:134)
    at io.netty.buffer.PoolArena.allocate(PoolArena.java:126)
    at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:394)
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
    at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
    at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:140)
    at io.netty.handler.codec.dns.DatagramDnsQueryEncoder.allocateBuffer(DatagramDnsQueryEncoder.java:82)
    at io.netty.handler.codec.dns.DatagramDnsQueryEncoder.encode(DatagramDnsQueryEncoder.java:60)
    at io.netty.handler.codec.dns.DatagramDnsQueryEncoder.encode(DatagramDnsQueryEncoder.java:33)
    at io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:89)
    ... 52 more

This is specifically happening with the VertX version 4.2.4 async Postgresql client which is using the Netty async http engine.

The specific version of java is:

java --version
openjdk 11.0.13 2021-10-19
OpenJDK Runtime Environment 18.9 (build 11.0.13+8)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.13+8, mixed mode, sharing)

Are there any JVM or Docker or Kubernetes parameters I should try tweaking to fix this problem?

Solution

  • The task manager memory layout and configuration options are described in https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/memory/mem_setup_tm/.