Search code examples
csvapache-sparkapache-spark-2.3

Quotes not displayed in CSV output file

Initial data is in Dataset<Row> and I am trying to write to csv file an each cell value to be placed in quotes.

result.coalesce(1).write()
            .option("delimiter", "|")
            .option("header", "true")
            .option("nullValue", "")
            .option("quoteMode", "ALL_NON_NULL")
            .csv(Location);

Expected output:

null
"London"|"UK"
"Delhi"|"India"
"Moscow"|"Russia"

Current Output:

null
London|UK
Delhi|India
Moscow|Russia

Spark version is 2.3

Solution

  • "quoteMode" is an option of databrick's CSV writer. Here you are using spark's built in CSV writer which does not support that option. Have a look at this page for the available options.

    In your case, the option you are looking for is .option("quoteAll", true).