How to properly get `next` to use use overridden instance method `__next__`?

Question

Consider the following snippet.

class A:
    def __next__(self):
        return 2

a = A()
print(next(a),a.__next__()) # prints "2,2" as expected
a.__next__ = lambda: 4
print(next(a),a.__next__()) # prints "2,4". I expected "4,4"

Clearly, the property __next__ is updated by the patching, but the inbuilt next function does not resolve that.

The python 3 docs docs on the python datamodel that says

For instance, if a class defines a method named __getitem__(), and x is an instance of this class, then x[i] is roughly equivalent to type(x).__getitem__(x, i).

From this, I came up with a hack as below

class A:
    def next_(self):
        return 2

    def __next__(self):
        return self.next_()

a = A()
print(next(a),a.__next__()) # 2,2
a.next_ = lambda: 4
print(next(a),a.__next__()) # 4,4

The code works, but at the expense of another layer of indirection via another next_-method.

My question is: What is the proper way to monkey-patch the __next__ instance method? What is the rationale behind this design in python?

Answer 1

You can't. Special methods are special they cannot be overridden at the instance level. Period. If you want to "customize" the instance behaviour the correct way to do it is to simply have a proper implementation instead of a bogus implementation that you swap at runtime. Change the value instead of the method.

The rationale can be found in The History of Python - Adding Support for User-defined Classes at the end of following section:

Special Methods

As briefly mentioned in the last section, one of my main goals was to keep the implementation of classes simple. In most object oriented languages, there are a variety of special operators and methods that only apply to classes. For example, in C++, there is a special syntax for defining constructors and destructors that is different than the normal syntax used to define ordinary function and methods.

I really didn't want to introduce additional syntax to handle special operations for objects. So instead, I handled this by simply mapping special operators to a predefined set of "special method" names such as __init__ and __del__. By defining methods with these names, users could supply code related to the construction and destruction of objects.

I also used this technique to allow user classes to redefine the behavior of Python's operators. As previously noted, Python is implemented in C and uses tables of function pointers to implement various capabilities of built-in objects (e.g., “get attribute”, “add” and “call”). To allow these capabilities to be defined in user-defined classes, I mapped the various function pointers to special method names such as __getattr__, __add__, and __call__. There is a direct correspondence between these names and the tables of function pointers one has to define when implementing new Python objects in C.

In summary: types defined in C have a structure that contains pointers to special methods. Guido wanted to keep consistency with types defined in Python and so their special methods end up being used at the class level.

Could the implementation always follow the lookup order? Yes... at a huge cost in speed, since now even the C code would have to first perform a dictionary lookup on the instance to ensure whether or not a special method is defined and call that. Given that special methods are called often, especially for built-in types, it makes sense to just have a direct pointer to the function in the class. The behaviour of the python side is just consistent with this.

Python was never bright in the performance sector. Your suggested implementation would run extremely slowly, especially 20 years ago when it was design on way less powerful machines and when JITs were extremely rare and not so well understood (compared to the present).