Search code examples
pythonfunctionscopeclosuresfunction-binding

itertools.product function creation producing unexpected results


I'm having a little trouble understanding the results of the below snippet, and think it's because I'm confused about function binding. Why would the following snippets produce different results?

import itertools

def make_funcs(lst):
    for val in lst:
        def f():
            return sum(1 for i in range(10) if i > val)
        f.func_name = ">" + str(val)
        yield f
## examples:
for f in make_funcs(range(2)):
    print(f.func_name, f())
## prints:
>0 9
>1 8

## works as expected:
for f in make_funcs(range(2)):
    for g in make_funcs(range(2)):
        print(f.func_name, g.func_name, f() + g())

## prints:
>0 >0 18
>0 >1 17
>1 >0 17
>1 >1 16

while on the other hand:

## provides results that are counter-intuitive (to me, at least)
for f, g in itertools.product(make_funcs(range(2)), make_funcs(range(2))):
    print(f.func_name, g.func_name, f() + g())

## prints:
>0 >0 16
>0 >1 16
>1 >0 16
>1 >1 16

It appears to me that it's only grabbing/using/binding the last variable in each implicit for-loop for the calculation, even though it's grabbing the correct variable for the function names.

What am I missing about scoping or function definitions or closures (or whatever) that's causing these results?

NB: if any of the tags I put on this question are irrelevant, feel free to remove them - I put them all because I'm not sure what the issue is.


Solution

  • All functions still reference the variable val.

    def make_funcs(lst):
        a = []
        for val in lst:
            def f():
                return sum(1 for i in range(10) if i > val)
            f.func_name = ">" + str(val)
            a.append(f)
        return a
    

    Results in all prints being "counter intuitive".

    def make_funcs(lst):
        a = []
        for val in lst:
            def f():
                return sum(1 for i in range(10) if i > val)
            f.func_name = ">" + str(val)
            a.append(f)
        val = 10
        return a
    

    Always results in 0's.

    However, because you use a generator, the value of val is only changed after it has been used, so all seems well. When using itertools.product, the docs says it does this:

    def product(*args, repeat=1):
        # product('ABCD', 'xy') --> Ax Ay Bx By Cx Cy Dx Dy
        # product(range(2), repeat=3) --> 000 001 010 011 100 101 110 111
        pools = [tuple(pool) for pool in args] * repeat
        result = [[]]
        for pool in pools:
            result = [x+[y] for x in result for y in pool]
        for prod in result:
            yield tuple(prod)
    

    Which means it first iterates over both generators (effectively changing the value of val four times) and only then calculates the results.

    This all happens because val is defined in the scope of make_funcs (and not in the scope of f), so if a second call to the generator changes the value of val, all functions reference the new value.

    Edit: Please also read the answer by @newacct