Understanding a technique to make class-based decorators support instance methods

Question

I recently came across a technique in the Python decorator library's memoized decorator which allows it to support instance methods:

import collections
import functools


class memoized(object):
    '''Decorator. Caches a function's return value each time it is called.
    If called later with the same arguments, the cached value is returned
    (not reevaluated).
    '''
    def __init__(self, func):
        self.func = func
        self.cache = {}

    def __call__(self, *args):
        if not isinstance(args, collections.Hashable):
        # uncacheable. a list, for instance.
        # better to not cache than blow up.
            return self.func(*args)
        if args in self.cache:
            return self.cache[args]
        else:
            value = self.func(*args)
            self.cache[args] = value
            return value

    def __repr__(self):
        '''Return the function's docstring.'''
        return self.func.__doc__

    def __get__(self, obj, objtype):
        '''Support instance methods.'''
        return functools.partial(self.__call__, obj)

The __get__ method is, as explained in the doc string, where the 'magic happens' to make the decorator support instance methods. Here are some tests showing that it works:

import pytest

def test_memoized_function():
    @memoized
    def fibonacci(n):
        "Return the nth fibonacci number."
        if n in (0, 1):
            return n
        return fibonacci(n-1) + fibonacci(n-2)

    assert fibonacci(12) == 144

def test_memoized_instance_method():
    class Dummy(object):
        @memoized
        def fibonacci(self, n):
            "Return the nth fibonacci number."
            if n in (0, 1):
                return n
            return self.fibonacci(n-1) + self.fibonacci(n-2)            

    assert Dummy().fibonacci(12) == 144

if __name__ == "__main__":
    pytest.main([__file__])

What I'm trying to understand is: how does this technique work exactly? It seems to be quite generally applicable to class-based decorators, and I applied it in my answer to Is it possible to numpy.vectorize an instance method?.

So far I've investigated this by commenting out the __get__ method and dropping into the debugger after the else clause. It seems that the self.func is such that it raises a TypeError whenever you try to call it with a number as input:

> /Users/kurtpeek/Documents/Scratch/memoize_fibonacci.py(24)__call__()
     23                         import ipdb; ipdb.set_trace()
---> 24                         value = self.func(*args)
     25                         self.cache[args] = value

ipdb> self.func
<function Dummy.fibonacci at 0x10426f7b8>
ipdb> self.func(0)
*** TypeError: fibonacci() missing 1 required positional argument: 'n'

As I understand from https://docs.python.org/3/reference/datamodel.html#object.get, defining your own __get__ method somehow overrides what happens when you (in this case) call self.func, but I'm struggling to relate the abstract documentation to this example. Can anyone explain this step by step?

Answer 1

As far as I can tell, When you use a descriptor to decorate an instance method(actually, an attribute), it defines the behavior of how to set, get and delete this attribute. There is a ref.

So in your example, memoized's __get__ defines how to get attribute fibonacci. In __get__, it pass obj to self.__call__ which obj is the instance. And the key to support instance method is to fill in argument self.

So the process is:

Assume there is an instance dummy of Dummy. When you access to dummy's attribute fibonacci, as it has been decorated by memoized. The value of attribute fibonacci is returned by memoized.__get__. __get__ accept two arguments, one is the calling instance(here is dummy) and another is its type. memoized.__get__ fill instance into self.__call__ in order to fill in self argument inside original method fibonacci.

To understand descriptor well, there is an example:

class RevealAccess(object):
    """A data descriptor that sets and returns values
       normally and prints a message logging their access.
    """

    def __init__(self, initval=None, name='var'):
        self.val = initval
        self.name = name

    def __get__(self, obj, objtype):
        print('Retrieving', self.name)
        return self.val

    def __set__(self, obj, val):
        print('Updating', self.name)
        self.val = val

>>> class MyClass(object):
...     x = RevealAccess(10, 'var "x"')
...     y = 5
...
>>> m = MyClass()
>>> m.x
Retrieving var "x"
10
>>> m.x = 20
Updating var "x"
>>> m.x
Retrieving var "x"
20
>>> m.y
5