Search code examples
csvparsingunivocity

Univocity parser - false delimiter autodetection when too little information given

I set the parser to detect the delimiters automatically

CsvParserSettings settings = new CsvParserSettings();
settings.detectFormatAutomatically();

I have only 1 single record : 47W2E2qxPs, http://usda.gov/mattis.html

What I got :

code: 47W2E2qxPshttp url: //usda.gov/mattis.html

I expected the delimiter to be , and not :

so my expected result would be 47W2E2qxPs and http://usda.gov/mattis.html .

Could I fix it in an elegant way?

Solution

  • Author of the library here. The detection process is a heuristic that uses statistics collected from multiple rows of part of your input. Therefore it depends a lot on the size of the input.

    Its purpose is to handle situations where you can't easily determine what is the CSV format - such as when users upload random files to you. Don't use the detection process if you already know what is the correct delimiter.

    In your case, one row of data is absolutely not enough to reliably detect the delimiter, especially if there are multiple symbols present. There is little you can do about it except for testing what was the detected delimiter before continuing:

    parser.beginParsing(new File("/path/to/your.csv"));
CsvFormat format = parser.getDetectedFormat();
//check if the format is sane.

    The next version (2.6.0) will include more options to assist the heuristic such as providing a set of allowed characters to be used as delimiters - which will probably help in your case.