I need to insert 3-4 million rows into a PostgreSQL database. This will be static lookup data.
We have various staging environments on the way to production. I have seen all the answers here on Stack Overflow about how we could do it not using Flyway, but has anybody got any experience with using Flyway when dealing with this quantity of data? Is it a non-starter? Flyway perfectly solves all of the problems we are facing.
Internet searching doesn't really turn up much for this so I am guessing that Flyway just isn't meant for dealing with this quantity of data but I would be interested if anybody has tried it and what they found out. If this is even remotely possible, how should we divide the import files size-wise for optimisation?
Actually it ended up being around 3 million records. We broke the files into 100,000 records per file so about 30 files. Flyway has no issue at all with this, it takes 30 seconds or so to run the scripts.
We're not sure we are going to use flyway to keep the records up to date, but for priming the data it has been the perfect solution