Perform Incremental Sqoop on table that contains joins?

Question

I have some very large tables that I am trying to sqoop from a Source System Data Warehouse into HDFS, but limited bandwidth to do so. I would like to only pull the columns I need, and minimize the run-time for getting the tables stood up.

The sqoop currently pulls something like this:

SELECT
    ColumnA,
    ColumnB,
    ....
    ColumnN
FROM
    TABLE_A
LEFT JOIN
    TABLE_B
ON
     ...
LEFT JOIN
    TABLE_N
....

Is It possible to perform an incremental sqoop, given that the data is stored in a star-schema format, and the dimensions could update independently of the facts?

Or, is the only solution to sqoop the entire table, for the columns that I need, incrementally, and perform the joins on the HDFS side?

Answer 1

For incremental imports you need to use --incremental flag. Please refer to below link for more info :-

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_incremental_imports

you need to specify —incremental to tell sqoop that you want an incremental load —check-column to specify which column is used for incremental sqooping and —last-value to say from which value you want to start sqooping the next load. This is just half the picture. There are more ways to do this.for eg. you can use —query option and your query would be like Select * from table where column > 123. This is basically the same thing. You would need to record the last/max value for the selected column and use it for next import.