Search code examples
mysqlapache-pigsqoopbigdata

Load data coming from MySQL into PIG using Sqoop


I'm trying to play with Sqoop and PIG, so almost all documentation I have seen says that Sqoop is used to get data from MySQL and put it within HDFS repositories, then this data would be reached using PIG. Instead of that, I would like to know if I can read directly from pig a MySQL table. In case guys that your answer will be yes, please could you please share an example script. thanks, I appreciate.


Solution

  • You can not directly access MySQL from PIG.

    You have got two options:

    • Use Sqoop to fetch data into HDFS and then analyze it using PIG scripts.

    • Write UDF's and create connection(s) to MySQL and fetch data and do analysis. This is more like a regular Java JDBC code.

    Option b, will be a terrible choice if your data is huge. Not sure on your requirement, but you are killing the beauty of Sqoop. Sqoop elegantly creates Mappers which in turn creates multiple connections to the database to pull data faster.

    Maybe you can share more details on why exactly you are trying this.