Search code examples
javatalend

Slow processing of a huge dataset in Talend

I am pulling data from a SQL Server stored procedure using tMSSqlSP which is returning results in RECORD SET format. After this, I am parsing the record set and using a tMap and then inserting the record in to an Oracle table using tOracleOutput.

Processing is very slow (30 records per second).

The SQL Server stored procedure returns 500k records and this slow processing takes ages to complete.

Is anyone aware of the process to make insertion faster on Oracle side?

Solution

  • Maybe you have to use bulk insertion in oracle : tOracleOutputBulk component. It's more complex to configure but if you insert in one table with no transaction it's a powerfull solution.