CSV from list of dictionaries with differing length and keys

Question

I have a list of dictionaries that I want to write to a csv file. The first dictionary is of a different length and has different keys than the following dictionaries.

dict_list = [{"A": 1, "B": 2}, {"C": 3, "D": 4, "E": 5}, {"C": 6, "D": 7, "E": 8}, ...]

How do I write this to a csv-file so that the file looks like this:

A B C D E
1 2 3 4 5
    6 7 8
    . . .

Answer 1

The problem is that you will need the full column set to write the header at the beginning of the file. But apart from that, csv.DictWriter is what you need:

# optional: compute the fieldnames:
fieldnames = set()
for d in dict_list:
    fieldnames.update(d.keys())
fieldnames = sorted(fieldnames)    # sort the fieldnames...

# produce the csv file
with open("file.csv", "w", newline='') as fd:
    wr = csv.DictWriter(fd, fieldnames)
    wr.writeheader()
    wr.writerows(dict_list)

And the produced csv will look like this:

A,B,C,D,E
1,2,,,
,,3,4,5
,,6,7,8

---

If you really want to combine rows with disjoint set of keys, you could do:

# produce the csv file
with open("file.csv", "w", newline='') as fd:
    wr = csv.DictWriter(fd, sorted(fieldnames))
    old = { k: k for k in wr.fieldnames }     # use old for the header line
    for row in dict_list:
        if len(set(old.keys()).intersection(row.keys())) != 0:
            wr.writerow(old)                  # common fields: write old and start a new row
            old = row
        old.update(row)                       # disjoint fields: just combine
    wr.writerow(old)                          # do not forget last row

You would get:

A,B,C,D,E
1,2,3,4,5
,,6,7,8