I am trying to read a UTF16-LE JSON file with a "document per line". I During the read file I get the following exception. I don't see a "lineSep" option in the portal. How does one set this?
Operation on target dataflow1 failed:
{"StatusCode":"DFExecutorUserError","Message":"Job failed due to reason: at Source 'source1': java.lang.IllegalArgumentException: requirement failed: The lineSep option must be specified for the UTF-16LE encoding","Details":"java.lang.IllegalArgumentException: requirement failed: The lineSep option must be specified for the UTF-16LE encoding\n\tat scala.Predef$.require(Predef.scala:224)\n\tat org.apache.spark.sql.catalyst.json.JSONOptionsInRead.checkedEncoding(JSONOptions.scala:153)\n\tat org.apache.spark.sql.catalyst.json.JSONOptions$$anonfun$34.apply(JSONOptions.scala:109)\n\tat org.apache.spark.sql.catalyst.json.JSONOptions$$anonfun$34.apply(JSONOptions.scala:109)\n\tat scala.Option.map(Option.scala:146)\n\tat org.apache.spark.sql.catalyst.json.JSONOptions.<init>(JSONOptions.scala:109)\n\tat org.apache.spark.sql.catalyst.json.JSONOptionsInRead.<init>(JSONOptions.scala:133)\n\tat org.apache.spark.sql.catalyst.json.JSONOptionsInRead.<init>(JSONOptions.scala:139)\n\tat org.apache.spark.sql.execution.datasources.json.JsonFileFormat.inferSchema(JsonFileFormat.scala:55)\n\tat org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource.scala:193)\n\tat org.apache.spark.sql.execution.datasources.DataSource$$anonfun$6.apply(DataSource."}
Thanks!
Ed
UTF-16LE JSON file encoding format is currently not supported in the Azure data factory.
You can go through this Microsoft document for all the supported encoding types in JSON dataset in the Azure data factory.