Load Multiple CSV files into database using Spring Batch

Question

I want to load multiple CSV files into mysql database at single table using Spring Batch. The path of the files are derived from the following method.

    public List<String> getFilePath() {
    String inputPath = "E:\\input";
    List<String> inputCSVPaths = new ArrayList<String>();
    Map<String, List<String>> inputInfo = new HashMap<String, List<String>>();

    File inputFolder = new File(inputPath);
    File[] inputFiles = inputFolder.listFiles();
    for (File file : inputFiles) {
        inputCSVPaths.add(file.getAbsolutePath());
    }
    inputInfo.put("Introduction", inputCSVPaths);
    List<String> inputFile = inputInfo.get("Introduction");
    System.out.println("Input File :"+inputFile);
    return inputFile;

}

There are total 3 CSV files. But it reads only onle file and inserts data of only that CSV file. Is there wrong in getting resources.

@Autowired
private FilePathDemo filePathDemo;

 @Bean
public MultiResourceItemReader<Introduction> multiResourceItemReader() throws IOException {

    MultiResourceItemReader<Introduction> multiReader = new MultiResourceItemReader<Introduction>();
    ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();

    Resource[] resources;
    String filePath = "file:";
    List<String> path = filePathDemo.getFilePath();
    for (String introPath : path) {
        System.out.println("File Path of the Introduction CSV :" + introPath);

        resources = patternResolver.getResources(filePath + introPath);
        multiReader.setResources(resources);
    }

        FlatFileItemReader<Introduction> flatReader = new FlatFileItemReader<Introduction>();
    multiReader.setDelegate(flatReader);
    flatReader.setLinesToSkip(1);
    flatReader.setLineMapper(new DefaultLineMapper<Introduction>() {
        {
            setLineTokenizer(new DelimitedLineTokenizer() {
                {
                    setNames(new String[] { "id", "name", "age", "phoneNo"});
                }
            });
            setFieldSetMapper(new BeanWrapperFieldSetMapper<Introduction>() {
                {
                    setTargetType(Introduction.class);
                }
            });
        }
    });

    flatReader.close();
    multiReader.close();

    return multiReader;
}

Answer 1

There are two issues with your configuration:

• You are reassigning the resources array with a single file in the for loop. Hence, the MultiResourceItemReader will be configured with only one file.
• You are calling the close method on the MultiResourceItemReader and the delegate FlatFileItemReader but you should not. Spring Batch will call those methods when the step is complete.

You can find an example of how to configure the MultiResourceItemReader here: https://docs.spring.io/spring-batch/4.0.x/reference/html/readersAndWriters.html#multiFileInput