Search code examples
sqoop

Sqoop - In incremental append where will the last value getting stored


The bellow Sqoop job works perfectly fine. Each time when I execute , the last value is appending when new data comes in MySQL. My question is

1) In which location the last-value is getting stored?

2) Can I browse the location of last value?

3) Can I change the location of last value?

sqoop job --create myjob3 \
-- import \
--connect "jdbc:mysql://ip-122-41-11-214:3306/sqoop" \
--username wikihadoop \
--password ABCD1234 \
--target-dir /user/sachin/inDataX3 \
--table sac01 \
--split-by age \
--num-mappers -1 \
--incremental append \
--check-column id \
--last-value 1

I executed this for sqoop

sqoop job --exec myjob3

Solution

  • 1) In which location the last-value is getting stored?

    It depends on how you are running your import job:

    • from the command line: last-value is printed on the screen as an output which you can use in next import. Sqoop will not manage it in this case. It's upto the user to keep track of it.
    • from a saved job: Sqoop uses it's private metastore (in /.sqoop/) or a shared metastore(as specified in sqoop-site.xml) to save job details.

    2) Can I browse the location of last value?

    • For shared metastore: you can use metastore browsers
    • For private merastore: you can try sqoop-metastore

    3) Can I change the location of last value?

    Yes. You can change the location of the metastore where last-value is stored.