Search code examples
javaspringspring-batchbatch-processing

Import large csv files spring

I have not used Spring Batch so far and i was wondering if this is maybe the time to break the ice.

I have large csv files with 10k to 30k lines that i need to import into database. Also i would need to do some processing of that data. Like checking some fields if they exist in the db (if not the row should be ignored). I never used Spring Batch so i would need to spend some time getting to know it so i can use it. But also i have a really strict deadline for this.

But is it really worth it for this kind of work? Since it would not be something like scheduled job witch would be done on daily, weekly or monthly basis. It would be done per need. Maybe once in a few months.

So is it "overkill" to use batch processing or it would be fine to just iterate line by line with some buffered reader?

Solution

  • If it is a one-off job for "just" 10k-20k lines, Spring Batch is complete overkill and you are better of writing something smaller, yourself.