Search code examples
pysparkapache-spark-sqlexport-to-csv

How to remove double quotes in csv generated after pyspark conversion

When I use pyspark to generate a csv file, the "null" str will be displayed in double quotes:

for example, the input is:

hello world

(null string) happy

the csv file generated is:

hello world

"" happy
you know, when use python to read this file, "" dosen't mean empty string.
So, is there anyway to remove double quotes in csv?
Thx

I try to use .option('nullValue', ''), but it seems not working.

Solution

  • .option('nullValue', None) would work