Search code examples
hivesqoopoozie

sqoop incremental import to hive table

I am trying to do a sqoop incremental import (lastmodified) to a hive table . It is working in sqoop 1.4.5 but not in sqoop 1.4.6 .

It is throwing the following error

**Log Upload Time: Mon Nov 13 17:57:58 +0530 2017

        Log Length: 246
      --incremental lastmodified option for hive imports is not supported. Please remove the parameter --incremental lastmodified.

Intercepting System.exit(1) Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]**

Solution

  • Sqoop 1.4.6 doesn't support incremental import to a hive table directly.

    However you can import the incremental data without using the --hive-import option and place the incremental data in the HDFS location which is mapped to the hive table using the --target-dir option.

    For example

    Consider there's a table sqoop_pract with location set to /user/hive/warehouse/sqoop_pract

    If you want to perform incremental import for this table you can use a query similar to the following

    sqoop import --connect jdbc:mysql://hostname/sample_db --username demo --table sqoop_pract --target-dir /user/hive/warehouse/sqoop_pract --incremental lastmodified --check-column Ts --last-value '2017-11-08 03:51:42' --append -P

    In the above command the --target-dir points to the hive table's location, this way you can achieve incremental import to a hive table using sqoop.