Search code examples
sqlapacheetldata-warehouseflume

Is there a native SQL source in Apache Flume?

I need to create a simple data warehouse. The data sources for the data warehouse are heterogeneous, thus I'm experimenting with Frameworks like Apache Flume for data collection. I went through the documentation but didn't find anything about SQL. (http://flume.apache.org/FlumeDeveloperGuide.html and http://flume.apache.org/FlumeUserGuide.html#flume-sources)

Question: Are there any (native) possibilities to connect an Apache Flume source to an SQL server?

Solution

  • Look into this project flume-ng-sql-source. Here are some examples as well.

    1. http://www.toadworld.com/platforms/oracle/w/wiki/11093.streaming-oracle-database-logs-to-hdfs-with-flume

    2. http://www.toadworld.com/platforms/oracle/w/wiki/11100.streaming-mysql-table-data-to-oracle-nosql-database-with-flume