Search code examples
parameter-passingloadingdatastageincremental-build

Datastage incremental charging

  • We would like to perform incremental loading in DataStage (in parallel environement). Exactly load only the delta between the previous load and the new one (for create, update, delete the records in DWH).

  • We would like to store the last key recovered during the previous load to be able to restart the request from the next record on a new loading.

  • We have already successfully used a parameter to filter the SQL load query at runtime. Unfortunately, we have not yet found the possibility to retrieve the last key (max (Key) - Aggregator?) And to store it in this parameter.

  • Which stage to use, to output a single value in the same parallel job, and then store to parameter ?

Any ideas ?

Thanks for your help.

Solution

  • Think about getting the max value from your target - it is most probably a database and a max() is easy to do.

    Check out my post about getting some data from the "stream" to a parameter