Search code examples
apache-sparkspark-structured-streamingspark-csv

Why I'm getting CSVHeaderChecker:69 - CSV header does not conform to the schema.?


When reading the csv data I'm getting the warning like that and no data is picked to the dataFrame batches.

The schema is exactly as exists in the csv. What could be the reason of the worning and the wrong behavior?


Solution

  • The reason was the different delimeter chars used in the header line and other data lines of the csv. Because of that the header was misinterpreted.