Search code examples
javaspring-bootspring-batchbatch-processingspring-batch-excel

Step freeze. PoiItemreader does not read excel file when the file is large. Any suggestion?


I tried to process an excel file using PoiItemreader in Spring Batch. The program runs successfully when the excel file is smaller or of normal size. However, when I tried to process the bigger file ( Bigger than 12MB). The file is not being read at all.

I have following questions:

  1. What is the limit of file size to use PoiItemreader?
  2. Will using MultiResourcePartioner work with this problem scenario?

Thank you very much.

Here is my code:

@Configuration
@EnableBatchProcessing
public class BatchConfiguration {

    @Autowired
    public JobBuilderFactory jobBuilderFactory;
    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Bean
    public Job csvProcessJob() throws FileNotFoundException {
        return jobBuilderFactory.get("csvProcessJob")
                .incrementer(new RunIdIncrementer())
                .flow(csvProcessStep())
                .end()
                .build();
    }

    @Bean
    public Step csvProcessStep() throws FileNotFoundException {
        return stepBuilderFactory.get("stepCSVprocess")
                .<String, Map<String, AttributeValue>> chunk(25)
                .reader(excelReader())
                .processor(processor())
                .writer(writer())
                .build();
    }

    @Bean
    public PoiItemReader excelReader() throws FileNotFoundException {
        PoiItemReader reader = new PoiItemReader();
        reader.setLinesToSkip(1);
        reader.setResource(new ClassPathResource("file_name.xls"));
        reader.setRowMapper(excelRowMapper());
        return reader;
    }

    private RowMapper<MetaData> excelRowMapper() {
        return new MetaDataRowMapper();
    }

    @Bean
    public ItemProcessor<MetaData,Map<String,AttributeValue>> processor() {
        return new MapProcessor();
    }

    @Bean
    public ItemWriter writer() {
        return new AWSwriter();
    }

Solution

  • Update for my question, I follow the link in the comment as M. Deinum post and be able to corporate that to my own custom itemreader. Now the program is running properly with the down side that it works only with .xlsx not .xls