Search code examples
hadoopsqoopflumebigdata

Clarification of Sqoop and Flume

I am very new to big data and i have little confusion regarding Sqoop and Flume

So i get that difference between the Sqoop and Flume

  • Sqoop is for transferring bulk data from RDBMS
  • Flume is for streaming of data such as log files

My confusion is because big data architecture i am looking at (which i have no virtual copy of) grouped structured data and its transferred by Sqoop and Unstructured streamed by Flume.

My question regard that is does that mean Flume is only for streaming?

What about high frequency data? and does Flume support transfer of unstructured data that are non-log files (i.e. audio, video) or would Sqoop be able to handle that?

Final question is can Sqoop work with federated data sources? if yes with both real and virtual?

Thanks,

Solution

  • Apache Flume is a distributed, reliable, and available system for efficiently collecting, aggregating and moving large amounts of log data from many different sources to a centralized data store.

    The use of Apache Flume is not only restricted to log data aggregation. Since data sources are customizable, Flume can be used to transport massive quantities of event data including but not limited to network traffic data, social-media-generated data, email messages and pretty much any data source possible.

    Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases(it imports data, transform the data in Hadoop MapReduce, and then export the data).

    Sqoop automates most of this process, relying on the database to describe the schema for the data to be imported. Sqoop uses MapReduce to import and export the data, which provides parallel operation as well as fault tolerance.

    Sqoop Vs Flume

    Source: sqoop-vs-flume-battle-of-the-hadoop

    Reference: INGESTION AND STREAMING

    Flume is efficient with streams and if you want to just dump data from RDBMS why not use sqoop?

    By high frequency data if you mean social media yes flume can handle it. Unstructured data yes, flume may handle that too.

    When to use?