Saving file information in Spring batch MultiResourceItemReader

Question

I have a directory having text files. I want to process files and write data into db. I did that by using MultiResourceItemReader.

I have a scenario like whenever file is coming, the first step is to save file info, like filename, record count in file in a log table(custom table).

Since i used MultiResourceItemReader, It's loading all files once and the code which i wrote is executing once in server startup. I tried with getCurrentResource() method but its returning null.

Please refer below code.

NetFileProcessController.java

@Slf4j
@RestController
@RequestMapping("/netProcess")
public class NetFileProcessController {

    @Autowired
    private JobLauncher jobLauncher;

    @Autowired
    @Qualifier("netFileParseJob")
    private Job job;

    @GetMapping(path = "/process")
    public @ResponseBody StatusResponse process() throws ServiceException {
        try {
            Map<String, JobParameter> parameters = new HashMap<>();
            parameters.put("date", new JobParameter(new Date()));

            jobLauncher.run(job, new JobParameters(parameters));
            return new StatusResponse(true);

        } catch (Exception e) {
            log.error("Exception", e);
            Throwable rootException = ExceptionUtils.getRootCause(e);
            String errMessage = rootException.getMessage();
            log.info("Root cause is instance of JobInstanceAlreadyCompleteException --> "+(rootException instanceof JobInstanceAlreadyCompleteException));
            if(rootException instanceof JobInstanceAlreadyCompleteException){
                log.info(errMessage);
                return new StatusResponse(false, "This job has been completed already!");
            } else{
                throw new ServiceException(errMessage);
            }
        }
    }
}

BatchConfig.java

@Configuration
@EnableBatchProcessing
public class BatchConfig {

    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    public void setJobBuilderFactory(JobBuilderFactory jobBuilderFactory) {
        this.jobBuilderFactory = jobBuilderFactory;
    }

    @Autowired
    StepBuilderFactory stepBuilderFactory;

    @Value("file:${input.files.location}${input.file.pattern}")
    private Resource[] netFileInputs;

    @Value("${net.file.column.names}")
    private String netFilecolumnNames;

    @Value("${net.file.column.lengths}")
    private String netFileColumnLengths;


    @Autowired
    NetFileInfoTasklet netFileInfoTasklet;

    @Autowired
    NetFlatFileProcessor netFlatFileProcessor;

    @Autowired
    NetFlatFileWriter netFlatFileWriter;

    @Bean
    public Job netFileParseJob() {
        return jobBuilderFactory.get("netFileParseJob")
                .incrementer(new RunIdIncrementer())
                .start(netFileStep())
                .build();
    }

    public Step netFileStep() {
        return stepBuilderFactory.get("netFileStep")
                .<NetDetailsDTO, NetDetailsDTO>chunk(1)
                .reader(new NetFlatFileReader(netFileInputs, netFilecolumnNames, netFileColumnLengths))
                .processor(netFlatFileProcessor)
                .writer(netFlatFileWriter)
                .build();
    }

}

NetFlatFileReader.java

    @Slf4j
    public class NetFlatFileReader extends MultiResourceItemReader<NetDetailsDTO> {

        public netFlatFileReader(Resource[] netFileInputs, String netFilecolumnNames, String netFileColumnLengths) {
            setResources(netFileInputs);
            setDelegate(reader(netFilecolumnNames, netFileColumnLengths));
        }

        private FlatFileItemReader<NetDetailsDTO> reader(String netFilecolumnNames, String netFileColumnLengths) {
            FlatFileItemReader<NetDetailsDTO> flatFileItemReader = new FlatFileItemReader<>();
            FixedLengthTokenizer tokenizer = CommonUtil.fixedLengthTokenizer(netFilecolumnNames, netFileColumnLengths);
            FieldSetMapper<NetDetailsDTO> mapper = createMapper();
            DefaultLineMapper<NetDetailsDTO> lineMapper = new DefaultLineMapper<>();
            lineMapper.setLineTokenizer(tokenizer);
            lineMapper.setFieldSetMapper(mapper);
            flatFileItemReader.setLineMapper(lineMapper);
            return flatFileItemReader;
        }

        /*
         * Mapping column data to DTO
        */
        private FieldSetMapper<NetDetailsDTO> createMapper() {
            BeanWrapperFieldSetMapper<NetDetailsDTO> mapper = new BeanWrapperFieldSetMapper<>();
            try {
                mapper.setTargetType(NetDetailsDTO.class);
            } catch(Exception e) {
                log.error("Exception in mapping column data to dto ", e);
            }
            return mapper;
        }

    }

I am stuck on this scenario, Any help appreciated

Answer 1

I don't think MultiResourceItemReader is appropriate in your case. I would run a job per file for all the reasons of making one thing do one thing and do it well:

• Your preparatory step will work by design
• It would be easier to run multiple jobs in parallel and improve your file ingestion throughput
• In case of failure, you would only restart the job for the failed file

EDIT: add an example

Resource[] netFileInputs = ... // same code that looks for file as currently in your reader
for (Resource netFileInput : netFileInputs) {
    Map<String, JobParameter> parameters = new HashMap<>();
    parameters.put("netFileInput", new JobParameter(netFileInput.getFilename()));

    jobLauncher.run(job, new JobParameters(parameters));
}