Talend ETL - Extract new records being inserted into the table

Question

Hi,

I have created a Talend ETL job which uses a query to a tmap to insert records into a table called "loan_grouping". I am using a lookup in the tmap to the "loan_grouping" table to pull through already existing records.

What I need is a way to extract only the newly created and inserted records. I have found guides online which show how to pull the rejected records using the Catch lookup inner join reject option in the tmap - but I need the opposite - I need to show the successful records.

The expression I have used in the tmap is:

!row1.GROUP_REFERENCE.equals(row3.GROUP_REFERENCE) 

But it still only gives me the rejected records. I've also attached a couple of screenshots. If anyone can help, I'd be really grateful!

Answer 1

You are confused between inner join and filter of tMap component, as i understood, you dont need a filter, just an inner join and their accepted and rejected data, so you need to delete the condition !row1.GROUP_REFERENCE.equals(row3.GROUP_REFERENCE) from the filter box at the top right. Since there is an inner join in this tMap, you cannot get all records in an output, we can have records that matches inner join and rejected records, so we can use tUnite after this tmap to merge the two outputs and get all data back or use the initial source:

                       lookup
                          |
tReplicate------(main)---tmap-------- (records that already have ref)
         \                  \
          \                  rejected (new records)
           \
            (all records)