How is the __class__ cell value set in class methods?

Question

Looking at the documentation of the super type in Python 3.5, it notes that super(…) is the same as super(__class__, «first argument to function»). To my surprise, I wrote a method that returned __class__ – and it actually worked:

>>> class c:
...   def meth(self): return __class__
... 
>>> c().meth()
<class '__main__.c'>

Apparently, __class__ is a free variable assigned by the closure of the function:

>>> c.meth.__code__.co_freevars
('__class__',)
>>> c.meth.__closure__
(<cell at 0x7f6346a91048: type object at 0x55823b17f3a8>,)

I'd like to know under what circumstances that free variable is associated in the closure. I know that if I assign a function to a variable as part of creating a class it doesn't happen.

>>> def meth2(self): return __class__
... 
>>> meth2.__code__.co_freevars
()

Even if I create a new class and as part of that creation assign some attribute to meth2, meth2 doesn't somehow magically gain a free variable that gets filled in.

That's unsurprising, because part of this appears to depend on the lexical state of the compiler at the time that the code is compiled.

I'd like to confirm that the conditions necessary for __class__ to be treated as a free variable are simply:

• A reference to __class__ in the code block; and
• The def containing the __class__ reference is lexically within a class declaration block.

I'd further like to understand what the conditions necessary for that variable getting filled in correctly are. It appears – at least from the Python 3.6 documentation – that something like type.__new__(…) is involved somehow. I haven't been able to understand for sure how type comes into play and how this all interacts with metaclasses that do not ultimately call type.__new__(…).

I'm particularly confused because I didn't think that at the time the namespace's __setattr__ method was used to assign the attribute containing the method to the method function (as it exists on the ultimately-constructed class object). I know that this namespace object exists because it was either constructed implicitly by the use of the class statement, or explicitly by the metaclass's __prepare__ method – but as best I can tell, the metaclass constructs the class object that populates __class__ after the function object is set as a value within the class namespace.

Answer 1

In the docs for Python’s data model, § 3.3.3.6 – “Creating the class object” – you will find the following:

[The] class object is the one that will be referenced by the zero-argument form of super(). __class__ is an implicit closure reference created by the compiler if any methods in a class body refer to either __class__ or super. This allows the zero argument form of super() to correctly identify the class being defined based on lexical scoping, while the class or instance that was used to make the current call is identified based on the first argument passed to the method.

…emphasis is mine. This confirms your two putative criteria for a __class__ closure happening: a “__class__” reference in the method def, which itself is defined inside a class statement.

But then, the next ¶ in “Creating the class object” goes on to say:

CPython implementation detail: In CPython 3.6 and later, the __class__ cell is passed to the metaclass as a __classcell__ entry in the class namespace. If present, this must be propagated up to the type.__new__ call in order for the class to be initialized correctly. Failing to do so will result in a RuntimeError in Python 3.8.

… emphasis is theirs. This means that if you are employing a metaclass with a __new__ method – in order to dictate the terms by which classes so designated are created – for example like e.g.:

class Meta(type):

    def __new__(metacls, name, bases, attributes, **kwargs):
        # Or whatever:
        if '__slots__' not in attributes:
            attributes['__slots__'] = tuple()

        # Call up, creating and returning the new class:
        return super().__new__(metacls, name,
                                        bases,
                                        attributes,
                                      **kwargs)

… that last super(…).__new__(…) call is effectively calling type.__new__(…). In real life, there might be some other ancestral “__new__(…)” methods that get called between here and there, if your metaclass inherits from other metaclasses (like, e.g. abc.ABCMeta). Effectively, though, inside your Meta.__new__(…) method, between the method entry point, the super(…).__new__(…) call, and return-ing the new class object, you can inspect or set the value of the eventual __class__ cell variable through attributes['__classcell__']†.

Now as for whether this is at all useful: I don’t know. I have been programming in python for ten years; I totally use metaclasses‡, like, absolutely all the time (for better or for worse); and in the course of doing so I have never done any of the following things:

1. reassigned a __class__ attribute;
2. inspected the __class__ cell variable of anything; nor
3. messed around with this supposed __classcell__ namespace entry, in like any capacity

… Naturally, your programming experience will be different from mine, who knows what one does. It is not that any one of those aforementioned stratagems are de facto problematic, necessarily. But I am no stranger to bending Python’s type systems and metaprogramming facilities to my whim, and these particular things have never presented themselves as partiuclarly useful, especially once you are working within the general context of metaclasses, and what they do.

By which I suppose I mean, tl;dr: you are on the cusp of figuring out the basics of metaclasses and what they can do – do press on and experiment, but do investigate the topic with depth as well as breath. Indeed!

---

† – In reading through code examples of this sort, you’ll often find what my snippet here calls the attributes dictionary referred to as namespace or ns, or similar. It’s all the same stuff.

‡ – …and ABCs and mixins and class decorators and __init_subclass__(…) and the abuse of __mro_entries__(…) for personal gain; et cetera, ad nauseum