Create output file for each of input file from directory of files in Spring Batch

Question

I have a directory of CSV files which contains transaction information.
I need to read each file , process some business logic (validate with DB to check valid account and transaction or not) and write the valid transactions into new output file.

Input: Tranx_100.csv, Tranx_101.csv, Tranx_102.csv....

Ouput: Tranx_100_output.csv, Tranx_101_output.csv, Tranx_102_output.csv....

Want to use spring batch for this. Any suggestions on how to implement this?

For each file as input, process, output are same - can i run them as part of 'Step' and repeat the step for each input file in a JOB?

Answer 1

Instead of looping on the same step or using multiple steps in a single job, I would use a job instance for each file (ie the file is an identifying job parameter) and launch jobs in parallel. This approach has multiple advantages:

• Fault-tolerance: In case of failure, only the failed file is re-processed
• Scalability: You can run multiple jobs in parallel easily
• Logging: logs will be separated by design
• And all the good reasons of making one thing do one thing and do it well