Quotation problems in results ( python )

Question

I hope you can help me with these, I create a script to extract some tables from a DB and after doing that, convert them to JSONL file.

My script work but when comes to the results ( JSONL ), some quotes " remove from the results and I dont know why, my script:

import pyodbc
import fileinput
import csv
import pandas as pd
import json
import os
import sys

conn = pyodbc.connect('Driver={SQL Server};'
                      'Server=Test;'
                      'UID=test;'
                      'PWD=test;'
                      'Database=TEST;'
                      'Trusted_Connection=no;')
cursor = conn.cursor()

query = "SELECT * FROM placeholder"


with open(r"D:\Test.txt") as file:
    lines = file.readlines()
    print(lines)


for user_input in lines:

    result = query.replace("placeholder", user_input)
    print(result)
    sql_query = pd.read_sql(result,conn)
    df = pd.DataFrame(sql_query)
    user_inputs =  user_input.strip("\n")
    filename = os.path.join('D:\\', user_inputs + '.csv')
    df.to_csv (filename, index = False, encoding='utf-8', sep = '~', quotechar = "`", quoting=csv.QUOTE_ALL)
    print(filename)
    filename_json = os.path.join('D:\\', user_inputs + '.jsonl')
    csvFilePath = (filename)
    jsonFilePath = (filename_json)
    print(filename_json)
    df.to_json(filename_json, orient = "records",  lines = bool, date_format = "epoch", double_precision = 15, force_ascii = False, date_unit = 'ms', default_handler = str)

dir_name = "D:\\"
test = os.listdir(dir_name)

for item in test:
    if item.endswith(".csv"):
        os.remove(os.path.join(dir_name, item))

cursor.close()
conn.close()

Now the results ( example ):

{"SucCod":1,"SucNom":"CENTRAL                  ","SucUsrMod":"aleos     ","SucFecMod":1537920000000,"SucHorMod":"11:30:21","SucTip":"S","SucBocFac":4,"SucCal":"SUTH               ","SucNro":1524,"SucPis":6,"SucDto":"    ","SucCarTel":"55   ","SucTel":52001}

And I spect ( example ):

{"SucCod":"1","SucNom":"CENTRAL                  ","SucUsrMod":"aleos     ","SucFecMod":"1537920000000","SucHorMod":"11:30:21","SucTip":"S","SucBocFac":"4","SucCal":"SUTH               ","SucNro":"1524","SucPis":"6","SucDto":"    ","SucCarTel":"55   ","SucTel":"52001"}

Now as you can see some data are missing the quotation "

After checking the CSV file they maintain the quotations so the problem is when converting to JSONL.

Do you have any idea about the switch I need to use to maintain the quotations ???

Thanks so much.

Kind regards.

Answer 1

If everything in your DataFrame must be a string, try using DataFrame.astype

df_o = df.astype(str)
df_o.to_json()

otherwise:

df_o = df.astype(object)
df_o.to_json()