Search code examples
hadoopconfigurationhdfshbase

Is there a maximum storage space configuration in HDFS or HBase?

Is there a maximum storage space configuration in HDFS or HBase?

I've found

  • dfs.data.dir: "Comma separated list of paths on the local filesystem of a DataNode where it should store its blocks" and

  • dfs.datanode.du.reserved: but it's for non DFS space reserved for HDFS

in the hadoop documentation

For HBase I've found some heap size configurations and compaction intervals, Memstore Flush Size, but none of these seem to regulate maximum size for a single node.

Is there any configuration for either HBase or HDFS that regulates how much space they will occupy in a single node?

(I am running tests on a single machine)

Solution

  • Generally speaking, dfs.data.dir will be a formatted volume that is mounted specifically for HDFS data. Therefore, the "maximum" is the number of physical SATA/USB/M.2 NVME connectors on the datanode's motherboard times the size of the largest hard-drives you can find.

    If not using dedicated volumes/devices, the max is still limited by the disk sizes, but dfs.datanode.du.reserved will leave space for the OS to run on its own.

    Neither is related to memory usage / heap space