generating functions in a loop using variables from the loop cause shadowing effects of variables

Question

I want to generate small functions in a loop that access variables from the loop. Then I want to compose and apply the functions all at once. An MWE looks lie this:

from functools import reduce

def compose(*funcs):
    return lambda x: reduce(lambda y, f: f(y), reversed(funcs), x)


def replace_values_by_type(text, values, types) -> str:

    def peek(x):
        print('peek:', x)
        return x

    replacements = [lambda txt: peek(peek(txt).replace(peek(val), f'<{peek(typ)}>')) for val, typ in zip(values, types)]
    replace = compose(*replacements)

    text_with_replacements = replace(text)

    print(values)
    print(types)
    print(text)
    print(text_with_replacements)
    print()

    return text_with_replacements



replace_values_by_type('this is a test sentence', ['is', 'test'], ['A', 'B'])

When I run this I expected to get "this <A> a <B> sentence". But Only the last pair of val and typ from the loop actually are used. So I guess some shadowing or overwriting happens. Can you explain this?

-> % py mwe.py 
peek: this is a test sentence
peek: test
peek: B
peek: this is a <B> sentence
peek: this is a <B> sentence
peek: test
peek: B
peek: this is a <B> sentence
['is', 'test']
['A', 'B']
this is a test sentence
this is a <B> sentence

Btw. to isolate the issue I also wrote the function like this:

def replace_values_by_type(text, values, types) -> str:

    replacements = []

    for val, typ in zip(values, types):
        def f(txt):
            return txt.replace(val, f'<{typ}>')

        replacements.append(f)

    text_with_replacements = text
    for f in replacements:
        text_with_replacements = f(text_with_replacements)

    return text_with_replacements


print(replace_values_by_type('this is a test sentence', ['is', 'test'], ['A', 'B']))

The problem remains the same.

Answer 1

All the closures created by the list comprehension are in the same variable scope, and there's just a single instance of the val and typ variables for the loop. When the closures are called later, the variables have their values from the last iteration.

You need to generate the closures in a unique scope for each iteration. One way to do this is to write a separate function that returns the closures, since every function establishes a new scope.

def replacer(val, typ):
    return lambda txt: peek(peek(txt).replace(peek(val), f'<{peek(typ)}>'))

replacements = [replacer(val, typ) for val, typ in zip(values, types)]