What is the differnce between HDFS and ADLS?

Question

I am confused about how azure data lake store in different from HDFS. Can anyone pls explain it in simple terms ?

Answer 1

• HDFS is a file system. HDFS stands for Hadoop Distributed File system. It is part of Apache Hadoop eco system. Read more on HDFS

• ADLS is a Azure storage offering from Microsoft. ADLS stands for Azure Data Lake Storage. It provides distributed storage file format for bulk data processing needs.

  • ADLS is having internal distributed file system format called Azure Blob File System(ABFS). In addition, it also provides similar file system interface API like Hadoop to address files and directories inside ADLS using URI scheme. This way, it is easier for applications using HDFS to migrate to ADLS without code changes. For clients, accessing HDFS using HDFS driver, similar experience is got by accessing ADLS using ABFS driver.

Azure Data Lake Storage Gen2 URI

The Hadoop Filesystem driver that is compatible with Azure Data Lake Storage Gen2 is known by its scheme identifier abfs (Azure Blob File System). Consistent with other Hadoop Filesystem drivers, the ABFS driver employs a URI format to address files and directories within a Data Lake Storage Gen2 capable account.

More on Azure Data Lake Storage

Hadoop compatible access: Data Lake Storage Gen2 allows you to manage and access data just as you would with a Hadoop Distributed File System (HDFS). The new ABFS driver is available within all Apache Hadoop environments, including Azure HDInsight, Azure Databricks, and Azure Synapse Analytics to access data stored in Data Lake Storage Gen2.

UPDATE also, read about Hadoop Compliant File System(HCFS) which ensures that distributed file system (like Azure Blob Storage) API meets set of requirements to satisfy working with Apache Hadoop ecosystem, similar to HDFS. More on HCFS