Python closures / variable assignment of decorated functions -

Question

I am trying to learn about closures and decorators in python.

I understand that, in my code below, variable fn has been assigned to a 'cell' object, which will itself reference the function object my_func once that is passed as an argument to function "outer" and outer is called.

I do not understand why, when fn is called from within function inner (the decorated function), it calls inner (ie the decorated function) rather than the 'original' undecorated my_func function (which is what was passed as an argument to "outer")

I had not anticipated that decorating my_func would cause the cell which i mentioned above to now reference a "different version" of my_func, which it what it appears from the console output is what has happened.

I therefore have two questions about my code below:

1. In the assignment my_func = outer(my_func), what is the "my_func" on the left handside of the assignment? I had understood it was a variable called "my_func" which masked the function my_func (but that the original function which was defined as my_func remained).
2. why does fn now appear to reference inner rather than my_func which is what was passed. Maybe it does not in fact do this and I am misunderstanding something.

Any help would be gratefully appreciated.

I have wrote the following code:

def my_func():
  print(f"This function's name is '{my_func.__name__}'. Its ID is {hex(id(my_func))}")
  return None

def outer(fn):
  def inner():
    print(f"The function that is about to be called is called '{fn.__name__}'. Its ID is {hex(id(fn))}")
    return fn() 
  return inner


print(f"I have created a function called '{my_func.__name__}'. Its ID is {hex(id(my_func))}")

my_func = outer(my_func)

print(f"The function has been decorated. The variable 'my_func' refers to a function object called '{my_func.__name__}'. Its ID is {hex(id(my_func))}\n")

my_func()

The following is printed to the console:

I have created a function called 'my_func'. Its ID is 0x7f3c040cbb50
The function has been decorated. The variable 'my_func' refers to a function object called 'inner'. Its ID is 0x7f3c040f4040

The function that is about to be called is called 'my_func'. Its ID is 0x7f3c040cbb50
This function's name is 'inner'. Its ID is 0x7f3c040f4040

What I was expecting is:

I have created a function called 'my_func'. Its ID is 0x7f3c040cbb50
The function has been decorated. The variable 'my_func' refers to a function object called 'inner'. Its ID is 0x7f3c040f4040

The function that is about to be called is called 'my_func'. Its ID is 0x7f3c040cbb50
The function that is about to be called is called 'my_func'. Its ID is 0x7f3c040cbb50

I recognize that it is my_func = outer(my_func) which is the cause of the 'issue', but I do not understand why. I had expected that the closure fn would continue to reference the "original" undecorated my_func function. I dont understand why assigning my_func to outer(my_func) appears to 'change' the object that fn refers to.

Answer 1

When you decorated the my_func function, it was replaced by the inner function.

After being decorated, it is normal to have __my_func__.name == "inner".

To adapt the function name, use the functools.wraps decorator. functools.wraps allows the metadata of the decorated function to be transferred to the enclosing function.

def outer(fn):
    @functools.wraps(fn)
    def inner():
        print(...)
        return fn() 
    return inner

More explanations:

In the statement:

def my_func():
    print(f"Name is '{my_func.__name__}'. ID is {hex(id(my_func))}")
    return None

you create a my_func module variable that holds a reference to a function object. {my_func.__name__} refers to this module variable. Reference pointed by my_func will be resolved each time the function is executed.

my_func = outer(my_func)

After this the variable my_func contains a reference to the newly created inner function. Illustratively, the my_func function has been replaced by the inner function.

When the inner function was created, it captured the my_func reference (closure) as the fn local variable. Reference pointed by fn will never change, and will remain forever the original function.

Each time outer is called, it create a new unique inner function.

When the last print is executed:

print(f"Name is {my_func.__name__}...")

my_func reference the new function so my_func.__name__ is "inner".

In the following example we can see that the reference hosted by my_func changes and the new function has captured a reference to the old one:

>>> def my_func():
...     pass
... 
>>> hex(id(my_func))
'0x7ffb218c0430'
>>> my_func = outer(my_func)
>>> hex(id(my_func))
'0x7ffb218c0310'
>>> my_func.__closure__
(<cell at 0x7ffb218b7c10: function object at 0x7ffb218c0430>,)