Search code examples
javacsvparsingjavabeansunivocity

univocity parsing into bean results in null value


I am trying to use the univocity csv parser to parse a csv file with more than 3 million rows into list of java beans. I set it up like in my example, but when I parse the csv every java bean has null attribute values. I played around with the settings but could not find out where my mistake is. These are the maven dependencies I use:

<dependency>
    <groupId>org.apache.commons</groupId>
    <artifactId>commons-lang3</artifactId>
    <version>3.8.1</version>
</dependency>
<dependency>
    <groupId>com.univocity</groupId>
    <artifactId>univocity-parsers</artifactId>
    <version>2.7.6</version>
</dependency>

This is my test class:

public class ParserTest {
    public List<OdsTx> start(File file) {
        BeanListProcessor<OdsTx> rowProcessor = new BeanListProcessor<OdsTx>(OdsTx.class);
        CsvParserSettings settings = new CsvParserSettings();
        settings.setDelimiterDetectionEnabled(true, ';');
        settings.setProcessor(rowProcessor);
        CsvParser parser = new CsvParser(settings);
        parser.parse(file);
        return rowProcessor.getBeans();
    }
    public static void main(String[] args) {
        String filename = "input/ods_TX.csv";
        File file = new File(filename);
        int testrow = 3;
        ParserTest test = new ParserTest();
        List<OdsTx> result = test.start(file);
        System.out.println("result size: " + result.size());
        System.out.println(result.get(testrow).toString());
    }

}

And this is my bean:

public class OdsTx {
    @Parsed(index = 0)
    private String CARDID;
    @Parsed(index = 1)
    private String ACCEPTANCEDATE;
    @Parsed(index = 2)
    private String AMOUNT;
    @Override
    public String toString() {
        return ToStringBuilder.reflectionToString(this, ToStringStyle.MULTI_LINE_STYLE);
    }
    public String getCARDID() {
        return CARDID;
    }
    public void setCARDID(String cARDNO) {
        CARDID = cARDNO;
    }
    public String getACCEPTANCEDATE() {
        return ACCEPTANCEDATE;
    }
    public void setACCEPTANCEDATE(String aCCEPTANCEDATE) {
        ACCEPTANCEDATE = aCCEPTANCEDATE;
    }
    public String getAMOUNT() {
        return AMOUNT;
    }
    public void setAMOUNT(String aMOUNT) {
        AMOUNT = aMOUNT;
    }
}

This is my csv file:

CARDID;ACCEPTANCEDATE;AMOUNT
12345168852;2018-01-01-07.56.29.000000;900
1234100080716;2018-01-01-09.19.26.000000;1000
1234100087256;2018-01-01-09.32.53.000000;1000
1234100087256;2018-01-01-09.33.03.000000;1000
12345199915;2018-01-01-09.41.44.000000;200
12345199915;2018-01-01-09.41.46.000000;200

My result using my toString() method looks always like this:

result size: 6
de.westlotto.connect.mehrfach.model.csv.OdsTx@4b9af9a9[
  CARDNO=<null>
  ACCEPTANCE_DATE=<null>
  AMOUNT=<null>
]

EDIT:

I had an error in my file path. Now I get the following error:

Exception in thread "main" com.univocity.parsers.common.TextParsingException: java.lang.ArrayIndexOutOfBoundsException - 512
Hint: Number of columns processed may have exceeded limit of 512 columns. Use settings.setMaxColumns(int) to define the maximum number of columns your input can have
Ensure your configuration is correct, with delimiters, quotes and escape sequences that match the input format you are trying to parse
Parser Configuration: CsvParserSettings:
    Auto configuration enabled=true
    Autodetect column delimiter=false
    Autodetect quotes=false
    Column reordering enabled=true
    Delimiters for detection=null
    Empty value=null
    Escape unquoted values=false
    Header extraction enabled=false
    Headers=null
    Ignore leading whitespaces=true
    Ignore leading whitespaces in quotes=false
    Ignore trailing whitespaces=true
    Ignore trailing whitespaces in quotes=false
    Input buffer size=1048576
    Input reading on separate thread=true
    Keep escape sequences=false
    Keep quotes=false
    Length of content displayed on error=-1
    Line separator detection enabled=false
    Maximum number of characters per column=4096
    Maximum number of columns=512
    Normalize escaped line separators=true
    Null value=null
    Number of records to read=all
    Processor=com.univocity.parsers.common.processor.BeanListProcessor
    Restricting data in exceptions=false
    RowProcessor error handler=null
    Selected fields=field selection: [0, 1, 2]
    Skip bits as whitespace=true
    Skip empty lines=true
    Unescaped quote handling=nullFormat configuration:
        CsvFormat:
            Comment character=#
            Field delimiter=;
            Line separator (normalized)=\n
            Line separator sequence=\n
            Quote character="
            Quote escape character="
            Quote escape escape character=null

Solution

  • Author of the lib here. You didn't post what input you are working with, but I assume your troubles may be coming from settings.detectFormatAutomatically(';');.

    Check what format was detected with parser.getDetectedFormat(). It may have detected the incorrect delimiter.

    In an unrelated note, as you are mapping your attributes to fixed positions and not header names, you don't need that @Headers annotation. You also don't need the @Trim annotations in every attribute as the parser by default trims all values for you.