Search code examples
pentahodata-migrationpentaho-spoonpentaho-data-integration

what is the maximum size of data that pentaho can migrate

I am using Pentaho data integration tool for migration of database

Pentaho data integration
version: 7.0
Build date: Nov 5 2016

i have migrated data upto
25mb of data from ms sql server to mysql.
I just wanted to know what is the max i can migrate using Pentaho.
And when i will get memory out of bound error

Solution

  • PDI uses a streaming model, where new records are only input when the buffers of the next steps allow, so for most operations there is no limit. I regularly transfer many GBs of data without issues.

    There are some steps that logically need to read all the rows into memory, but then you're doing more advanced transformations than just migration.

    One thing you might run into when doing very large transfers in a single run is idle timeouts. For example if the destination MySQL is overloaded by the writes, PDI might wait long before requesting the next 10,000 rows from MS SQL and either the MS SQL or a firewall inbetween resets the connection. You can solve this by writing out to temporary files first (very fast) and then loading from file to MySQL using the bulk loader step.