How does buffer size affect NIO Channel performance?

Question

I was reading Hadoop IPC implementation. https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java

/**
 * When the read or write buffer size is larger than this limit, i/o will be 
 * done in chunks of this size. Most RPC requests and responses would be
 * be smaller.
 */
private static int NIO_BUFFER_LIMIT = 8*1024; //should not be more than 64KB.

/**
 * This is a wrapper around {@link WritableByteChannel#write(ByteBuffer)}.
 * If the amount of data is large, it writes to channel in smaller chunks. 
 * This is to avoid jdk from creating many direct buffers as the size of 
 * buffer increases. This also minimizes extra copies in NIO layer
 * as a result of multiple write operations required to write a large 
 * buffer.  
 *
 * @see WritableByteChannel#write(ByteBuffer)
 */
private int channelWrite(WritableByteChannel channel, 
                         ByteBuffer buffer) throws IOException {

  int count =  (buffer.remaining() <= NIO_BUFFER_LIMIT) ?
               channel.write(buffer) : channelIO(null, channel, buffer);
  if (count > 0) {
    rpcMetrics.incrSentBytes(count);
  }
  return count;
}


/**
 * This is a wrapper around {@link ReadableByteChannel#read(ByteBuffer)}.
 * If the amount of data is large, it writes to channel in smaller chunks. 
 * This is to avoid jdk from creating many direct buffers as the size of 
 * ByteBuffer increases. There should not be any performance degredation.
 * 
 * @see ReadableByteChannel#read(ByteBuffer)
 */
private int channelRead(ReadableByteChannel channel, 
                        ByteBuffer buffer) throws IOException {

  int count = (buffer.remaining() <= NIO_BUFFER_LIMIT) ?
              channel.read(buffer) : channelIO(channel, null, buffer);
  if (count > 0) {
    rpcMetrics.incrReceivedBytes(count);
  }
  return count;
}

The logic is, If the buffer is small, it'll read/write channel one time. If buffer is large, it'll do it many times, and every time read/write 8kb.

I don't understand the javadocs and why it does this way. Why "This is to avoid jdk from creating many direct buffers as the size of buffer increases."? Does big buffer size affect read performance as well?

I understand how buffer size affects FileInputStream performance (link). But here is SocketChannel. So it's unrelated.

Answer 1

Good question. sun.nio.ch.IOUtil is used while writing in channel, and it has the following lines in its write(..) function

int var7 = var5 <= var6?var6 - var5:0;
ByteBuffer var8 = Util.getTemporaryDirectBuffer(var7);

Here is Util.getTemporaryDirectBuffer

static ByteBuffer getTemporaryDirectBuffer(int var0) {
    Util.BufferCache var1 = (Util.BufferCache)bufferCache.get();
    ByteBuffer var2 = var1.get(var0);
    if(var2 != null) {
        return var2;
    } else {
        if(!var1.isEmpty()) {
            var2 = var1.removeFirst();
            free(var2);
        }

        return ByteBuffer.allocateDirect(var0);
    }
}

And under a heavy load and when int var0 is in a big range it would create lots of new buffers and free(..) the old ones. Because the bufferCache has limited length (equals to IOUtil.IOV_MAX which is defined in system config. On modern Linux systems, the limit is 1024) and wouldn't store buffers of every length.
I think this is meant in This is to avoid jdk from creating many direct buffers as the size of buffer increases..