Search code examples
pythonjsonpandasfor-loopinfluxdb

Convert CSV file data into JSON format using Python


I have few data in a CSV file. I wanted to convert this data into json format. By using the below code i am able to convert only last line of my CSV file, but not all rows of CSV data. Also in my csv file there are many columns but script is considering only column 3 values.

I could see that the change is required in for loop, Tried many ways but didnt succeeded can you please help me in resolving this issue?

My script is below -

import pandas as pd
from influxdb import InfluxDBClient

client = InfluxDBClient(host='localhost', port=8086)
client.switch_database('csvdata')

file_path = r'/home/ec2-user/influxdb-1.4.2-1/LEGO_throughput.csv'

csvReader = pd.read_csv(file_path)

#print(csvreader.shape)
#print(csvreader.columns)

for row_index, row in csvReader.iterrows():
    tags = row[0]
    fieldvalue = row[2]
    json_body = [
    {
        "measurement": "LEGO_throughput",
        "tags": {
                "Reference": tags
        },
        "fields": {
            "value": fieldvalue
        }
    }
        ]

client.write_points(json_body)

and my CSV data -

series(eventTimestamp),count(*),"percentile(responseTime, 95)",avg(responseTime)
2020-07-17T01:17:00+01:00,81,739,444.9753086
2020-07-17T01:18:00+01:00,784,2600,809.3762755
2020-07-17T01:19:00+01:00,3127,2825,1316.033259
2020-07-17T01:20:00+01:00,6348,2908,1421.663674


Solution

  • I used your implementation and changed it a bit. I am not sure if it's exactly what you wanted to achieve

    import pandas as pd
    import json
    
    file_path = r'./LEGO_throughput.csv'
    
    csvReader = pd.read_csv(file_path)
    
    #print(csvreader.shape)
    #print(csvreader.columns)
    json_body = []
    for row_index, row in csvReader.iterrows():
        tags = row[0]
        fieldvalue = row[2]
        json_body += [
        {
            "measurement": "LEGO_throughput",
            "tags": {
                    "Reference": tags
            },
            "fields": {
                "value": fieldvalue
            }
        }
            ]
    with open("res.json", "w") as res:
        json.dump(json_body, res)