Search code examples
pythonlistdictionaryappenddump

Is there a way to append a dumped dictionary or a dumped object in general using python?

I have a dictionary let say:

d = {"a": 1, "b": 2, "c": 3 }

All I want to do is to dump this dictionary into a file and then append it with data produced by my program little by little. I don't want to have the dictionary data in memory during the running of the program. By the end I want to be able to load this file into a python structure from another program.

Is there a solution to do this using pickle or json or another python module? I know that you can't append to a dumped object in a pickle-file.

I know that I can do something like this:

d = pickle.load(open(fileName, 'rb'))
d.update(dictWithNewValues)
pickle.dump(d, open(fileName, 'wb'))

as described here. But I want to do it progressively.. I understand that this may be difficult to do that way because dictionary has unique keys, so to be append it the current keys should be known to avoid conflicts (entries with same keys..).

So, is there a way to do it in a list or another python object?

Solution

  • Assuming the keys of your dictionary are all strings, the simplest option I see is Python's shelve module. A shelve behaves like a persistent dictionary, and any key can be updated.

    Another option is the anydbm module, which also requires all values to be strings.