I am transferring data from table in one schema to the table in another schema. I also need to record rows that get rejected in the process, due to one of the many reasons, for example, constraint of NOT NULL failing.
When I attach the reject link with the output component, the transfer speed decreases drastically to 2-3 rows per second. On the other hand, without reject link, I can use "Extend Insert" option and the speed increases to 400 rows per second.
How can I capture rejected rows without compromising on the performance?
I agree with the methodology suggested above by @Corentin . I came up with an effective solution for my problem using the same idea. Instead of letting DB return with an error for each row in case of breaking of constraint, I am checking for rejection within the Talend, using the component tSchemaComplianceCheck, which contains schema exactly the same as the destination table & carrying out rejection link from this component.