How does Python's super() actually work, in the general case?

Question

There are a lot of great resources on super(), including this great blog post that pops up a lot, as well as many questions on Stack Overflow. However I feel like they all stop short of explaining how it works in the most general case (with arbitrary inheritance graphs), as well as what is going on under the hood.

Consider this basic example of diamond inheritance:

class A(object):
    def foo(self):
        print 'A foo'

class B(A):
    def foo(self):
        print 'B foo before'
        super(B, self).foo()
        print 'B foo after'

class C(A):
    def foo(self):
        print 'C foo before'
        super(C, self).foo()
        print 'C foo after'

class D(B, C):
    def foo(self):
        print 'D foo before'
        super(D, self).foo()
        print 'D foo after'

If you read up on Python's rules for method resolution order from sources like this or look up the wikipedia page for C3 linearization, you will see that the MRO must be (D, B, C, A, object). This is of course confirmed by D.__mro__:

(<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, <type 'object'>)

And

d = D()
d.foo()

prints

D foo before
B foo before
C foo before
A foo
C foo after
B foo after
D foo after

which matches the MRO. However, consider that above super(B, self).foo() in B actually calls C.foo, whereas in b = B(); b.foo() it would simply go straight to A.foo. Clearly using super(B, self).foo() is not simply a shortcut for A.foo(self) as is sometimes taught.

super() is then obviously aware of the previous calls before it and the overall MRO the chain is trying to follow. I can see two ways this might be accomplished. The first is to do something like passing the super object itself as the self argument to the next method in the chain, which would act like the original object but also contain this information. However this also seems like it would break a lot of things (super(D, d) is d is false) and by doing a little experimenting I can see this isn't the case.

The other option is to have some sort of global context that stores the MRO and the current position in it. I imagine the algorithm for super goes something like:

1. Is there currently a context we are working in? If not, create one which contains a queue. Get the MRO for the class argument, push all elements except for the first into the queue.
2. Pop the next element from the current context's MRO queue, use it as the current class when constructing the super instance.
3. When a method is accessed from the super instance, look it up in the current class and call it using the same context.

However, this doesn't account for weird things like using a different base class as the first argument to a call to super, or even calling a different method on it. I would like to know the general algorithm for this. Also, if this context exists somewhere, can I inspect it? Can I muck with it? Terrible idea of course, but Python typically expects you to be a mature adult even if you're not.

This also introduces a lot of design considerations. If I wrote B thinking only of its relation to A, then later someone else writes C and a third person writes D, my B.foo() method has to call super in a way that is compatible with C.foo() even though it didn't exist at the time I wrote it! If I want my class to be easily extensible I will need to account for this, but I am not sure if it is more complicated than simply making sure all versions of foo have identical signatures. There is also the question of when to put code before or after the call to super, even if it does not make any difference considering B's base classes only.

Answer 1

super() is then obviously aware of the previous calls before it

It's not. When you do super(B, self).foo, super knows the MRO because that's just type(self).__mro__, and it knows that it should start looking for foo at the point in the MRO immediately after B. A rough pure-Python equivalent would be

class super(object):
    def __init__(self, klass, obj):
        self.klass = klass
        self.obj = obj
    def __getattr__(self, attrname):
        classes = iter(type(self.obj).__mro__)

        # search the MRO to find self.klass
        for klass in classes:
            if klass is self.klass:
                break

        # start searching for attrname at the next class after self.klass
        for klass in classes:
            if attrname in klass.__dict__:
                attr = klass.__dict__[attrname]
                break
        else:
            raise AttributeError

        # handle methods and other descriptors
        try:
            return attr.__get__(self.obj, type(self.obj))
        except AttributeError:
            return attr

If I wrote B thinking only of its relation to A, then later someone else writes C and a third person writes D, my B.foo() method has to call super in a way that is compatible with C.foo() even though it didn't exist at the time I wrote it!

There's no expectation that you should be able to multiple-inherit from arbitrary classes. Unless foo is specifically designed to be overloaded by sibling classes in a multiple-inheritance situation, D should not exist.