How to export multiple dictionaries with some different keys into one csv file in Python?

Question

I'm trying to write multiple dictionaries with different fieldnames into one csv file. I'd like to have all dictionaries' keys on top of the csv file as a header and all their values in each row. But my code doesn't do this and it adds each dictionary's keys into a separate row and its values in the row below.

from bs4 import BeautifulSoup
import os
import re
import csv

for j in range(1,39):
    for i in rabge(1,10):


### calculate super_dict for each html pages
            super_dict.update(social_urls)
            super_dict.update(metadata)
            super_dict.update(project_name)
            super_dict.update(bc_dict)
            super_dict.update(ind_dict)
            super_dict.update(pos_dict)
            super_dict.update(likes_dict)
            super_dict.update(memprof_dict)
            super_dict.update(video_link)
            super_dict.update(details_dict) 
            with open('output.csv', 'a') as csvfile:`
                fieldnames = super_dict.keys()
                writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
                writer.writeheader()
                writer.writerow(super_dict)

Answer 1

The problem is that the code writes a single dictionary, which the DictWriter treats as a single row. An approach that should work is to creates the fieldnames from all the dict's keys and then write each dict as a separate row.

Something like this:

dicts = [social_urls, metadata, ...]
fieldnames = set()
# Update fieldnames with each dict's keys
# Use a set to filter duplicates.
fieldnames.update(*(d.keys() for d in dicts))
with open(myfile.csv, 'a', newline='') as f:
    writer= csv.DictWriter(f, fieldnames=fieldnames)
    for d in dicts:
        writer.writerow(d)