Search code examples
pythonpython-3.xperformancebenchmarkingpython-internals

Why is python dict creation from list of tuples 3x slower than from kwargs


There are a couple of ways to construct a dictionary in python, for example:

keyvals = [('foo', 1), ('bar', 'bar'), ('baz', 100)]

dict(keyvals)

and

dkwargs = {'foo': 1, 'bar': 'bar', 'baz': 100}

dict(**dkwargs)

When you benchmark these

In [0]: %timeit dict(keyvals)
667 ns ± 38 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

In [1]: %timeit dict(**dkwargs)
225 ns ± 7.09 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

you see that the first method is almost 3x slower than the second. Why is this?


Solution

  • dict(**kwargs) passes in a ready-made dictionary, so Python can just copy an already existing internal structure.

    A list of tuples, on the other hand, requires iteration, validation, hashing and slotting the results into a fresh, empty table. That's not nearly as fast.

    A Python dictionary is implemented as a hash table, and is grown dynamically as keys are added over time; they start small and as the need arises a new, larger hash table is constructed, data (keys, values and hashes) are copied across. That’s all invisible from Python code but resizing takes time. But when you use dict(**kwargs) (or dict(other_dict) CPython (the default Python implementation you used to test with) can take a shortcut: start with a hash table that’s big enough immediately. You can’t do that same trick with a sequence of tuples, because you can’t know up front if there won’t be duplicate keys in the sequence.

    For more details, see the C source code of the dict type, specifically dict_update_common implementation (which is called from dict_init()); this calls either PyDict_MergeFromSeq2() for the sequence-of-tuples case, or calls PyDict_Merge() when keyword arguments are passed in.

    The PyDict_MergeFromSeq2() function iterates over the sequence, tests each result to make sure there are two elements, then essentially calls .__setitem__(key, value) on the dictionary. This may need to resize the dictionary at some point!

    The PyDict_Merge() function (via dict_merge()) specifically detects if a regular dictionary was passed in, then executes a fast path that resizes the internal structures once, and then copies across the hashes and structure from the original dictionary directly using insertdict() calls (follow the override == 1 path, as override has been set to 1 when the target dictionary is empty, which is always the case for dict(**kwargs)). Just resizing once and using internal data directly is a lot faster, far less work needs to be done!

    All of this is an implementation detail specific to CPython. Other Python implementations such as Jython, IronPython and PyPy can make their own decisions on how the internals of the dict type work, and will show different performance differences for the same operations.