pythoncsvgoogle-cloud-platformgoogle-cloud-storage

CSV file formatting doesn't work for GCP BlobWriter


I have same data as list of dict and want to import these in a CSV file in a GCP bucket.

I'm importing these object as streaming:

blob = defaults.bucket.blob(filename)
csv_writer = BlobWriter(blob, content_type="text/csv")

for data in result.get('events'):
    _source = data.get('_source', {})
    csv_writer.write(json.dumps(_source, indent=2).encode('utf-8'))  

where _source is dict.

The actual result in the CSV file is JSON formatting, not CSV.

For example:

{           
  'key':'value'     
}{          
  'key':'value'         
}{      

Solution

  • import csv
    
    blob = defaults.bucket.blob(filename)
    csv_writer = BlobWriter(blob, content_type="text/csv")
    
    # Assuming '_source' is a dictionary and result.get('events') is a list of dictionaries
    field_names = ['key1', 'key2', 'key3']  # Replace with actual keys in your '_source' dictionaries
    
    # Write CSV header
    csv_writer.write(','.join(field_names).encode('utf-8') + b'\n')
    
    for data in result.get('events'):
        _source = data.get('_source', {})
        
        # Convert each dictionary to a CSV row
        csv_row = [str(_source.get(key, '')) for key in field_names]
        
        # Write CSV row to the file
        csv_writer.write(','.join(csv_row).encode('utf-8') + b'\n')