Search code examples
hadoophbasehdfsreplicationopentsdb

What is the Relationship between HDFS Block Size and Network Replication Traffic?

Our HBase cluster has a significant amount of network traffic relative to the amount of data we believe we are writing to it via OpenTSDB.

Relayed Bytes to OpenTSDB:

enter image description here

Network Throughput in Bytes (Y Axis is SI units, so not base2):

enter image description here

We use Cloudera, so the block is 128MB, would changing it to 64MB reduce network replication traffic? Are there other consequences to consider?

Solution

  • To answer plainly the question of will lowering block size reduce replication traffic, the answer is no.

    HDFS clients do not write entire block worth of data at once - they stream the writes in small checksummed packets of 64k-128k bytes at a time (configurable), building towards the entire block at the DataNode(s) in the pipeline.

    Lowering the block size will only have one effect: more blocks across more DataNodes. It will not alter the rates of transfer and the total network utilisation, which is entirely driven by the goal data write size and the replication factor and not by the HDFS logical block size attribute.