Pandas to_csv writing extra " on column containing raw json

Question

I'm attempting to save my dataframe as csv after processing my data.

One caveat is I have a column containing the 'raw json' of the file as well.

When pandas saves the file using to_csv(header=False), I get the following

1,2,"{""col_1"":""1"",""col_2"":""1""}"

My dataframe looks like this:

col_1	col_2	raw_json
1	1	{"col_1":1,"col_2":1}

I've tried adding the json col something like:

for i, row in df:
    i_val = row.to_json()
    df.at[i,'raw_json'] = i_val

Expected csv:

1,2,{"col_1":"1","col_2":"1"}

Answer 1

You could use something like this:

import csv
import pandas as pd

df.to_csv('output.csv', index=False, header=False, quoting=csv.QUOTE_NONE, sep=';')

As @pranav-hosangadi was explaining:

"CSV format uses quotes to escape fields that themselves contain the separator"

So when you set quoting=csv.QUOTE_NONE you disable that behavior and nothing will be quoted.

Important: Note that the separator of the csv will be ";" in this case, so you'll need to be sure that your fields not contains";" characters that could broke your csv