Search code examples
pythonpython-3.xlist-comprehensionpython-3.5iterable-unpacking

Unpacking generalizations

PEP 448 -- Additional Unpacking Generalizations allowed:

>>> LOL = [[1, 2], ['three']]
>>> [*LOL[0], *LOL[1]]
[1, 2, 'three']

Alright! Goodbye itertools.chain. Never liked you much anyway.

>>> [*L for L in LOL]
  File "<ipython-input-21-e86d2c09c33f>", line 1
    [*L for L in LOL]
    ^
SyntaxError: iterable unpacking cannot be used in comprehension

Oh. Why can't we have nice things?

Unfortunately there are syntax errors for all of them:

[*l for l in lists]    # for l in lists: result.extend(l)
{*s for s in sets}     # for s in sets: result.update(s)
{**d for d in dicts}   # for d in dicts: result.update(d)
(*g for g in gens)     # for g in gens: yield from g

Unpacking in a comprehension seems to be obvious and Pythonic, and there's a quite natural extension from shorthand "for-loop & append" to "for-loop & extend".

But since they've bothered to add that special error message, there was presumably a reason for disabling it. So, what's the problem with that syntax?

Solution

  • This is briefly explained in the PEP 448 which introduces unpacking generalizations:

    Earlier iterations of this PEP allowed unpacking operators inside list, set, and dictionary comprehensions as a flattening operator over iterables of containers:

    >>> ranges = [range(i) for i in range(5)]
>>> [*item for item in ranges]
[0, 0, 1, 0, 1, 2, 0, 1, 2, 3]

>>> {*item for item in ranges}
{0, 1, 2, 3}

    This was met with a mix of strong concerns about readability and mild support. In order not to disadvantage the less controversial aspects of the PEP, this was not accepted with the rest of the proposal.

    However, this may change in the future:

    This PEP does not include unpacking operators inside list, set and dictionary comprehensions although this has not been ruled out for future proposals.

    The PEP mentions "strong concerns about readability". I don't know the entire story, but the detailed discussions that led to this decision can certainly be found in the mailing list:

    Here is an ambiguous example if unpacking generalizations were to be allowed in list comprehension:

    [*t for t in [(1, 'a'), (2, 'b'), (3, 'c')]]

    According to one of the core developers, it would be surprising for the result to be [1, 'a', 2, 'b', 3, 'c'] and not [(1, 'a'), (2, 'b'), (3, 'c')].

    Since there was no formal consensus, it was simpler not to allow these special cases.