Search code examples
dockerhadoophdfsdocker-swarmhdp

HDFS on docker containers - is it possible?

we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform

we want to know if we can use the following HDFS components on docker containers as:

1. journal nodes 
2. ZK fail controller
3. NameNode ( HA with active namenode and standby namenode )

after some searching , we not found so much documentation about HDFS on docker containers

so I am assume that HDFS on docker containers isnt good implementation

I will happy to get more ideas about this?

Solution

  • Cloudera has no such documentation to my knowledge. They may be working on a future Kubernetes product offering, but that is just what I have heard.

    Yes, it is possible to use Docker, but it would not be supported by any major Hadoop vendor, and it would probably be very difficult to handle networking and storage requirements around that.

    MinIO would arguably be a better alternative than HDFS when using Docker