Why differences in class.__slots__ assignment via decorator vs. class body?

Question

I'm working on a decorator to implement some behaviors for an immutable class. I'd like a class to inherit from namedtuple (to have attribute immutability) and also want to add some new methods. Like this ... but correctly preventing new attributes being assigned to the new class.

When inheriting from namedtuple, you should define __new__ and set __slots__ to be an empty tuple (to maintain immutability):

def define_new(clz):
    def __new(cls, *args, **kwargs):
        return super(clz, cls).__new__(cls, *args, **kwargs)

    clz.__new__ = staticmethod(__new) # delegate namedtuple.__new__ to namedtuple
    return clz

@define_new
class C(namedtuple('Foo', "a b c")):
    __slots__ = () # Prevent assignment of new vars
    def foo(self): return "foo"

C(1,2,3).x = 123 # Fails, correctly

... great. But now I'd like to move the __slots__ assignment into the decorator:

def define_new(clz):
    def __new(cls, *args, **kwargs):
        return super(clz, cls).__new__(cls, *args, **kwargs)

    #clz.__slots__ = ()
    clz.__slots__ = (123) # just for testing

    clz.__new__ = staticmethod(__new)
    return clz

@define_new
class C(namedtuple('Foo', "a b c")):
    def foo(self): return "foo"

c = C(1,2,3)
print c.__slots__ # Is the (123) I assigned!
c.x = 456         # Assignment succeeds! Not immutable.
print c.__slots__ # Is still (123)

Which is a little surprising.

Why has moving the assignment of __slots__ into the decorator caused a change in behavior?

If I print C.__slots__, I get the object I assigned. What do the x get stored?

Answer 1

The code doesn't work because __slots__ is not a normal class property consulted at run-time. It is a fundamental property of the class that affects the memory layout of each of its instances, and as such must be known when the class is created and remain static throughout the its lifetime. While Python (presumably for backward compatibility) allows assigning to __slots__ later, the assignment has no effect on the behavior of existing or future instances.

How __slots__ is set

The value of __slots__ determined by the class author is passed to the class constructor when the class object is being created. This is done when the class statement is executed; for example:

class X:
    __slots__ = ()

The above statement is equivalent¹ to creating a class object and assigning it to X:

X = type('X', (), {'__slots__': ()})

The type object is the metaclass, the factory that creates and returns a class when called. The metaclass invocation accepts the name of the type, its superclasses, and its definition dict. Most of the contents of the definition dict can also be assigned later The definition dict contains directives that affect low-level layour of the class instances. As you discovered, later assignment to __slots__ simply has no effect.

Setting __slots__ from the outside

To modify __slots__ so that it is picked up by Python, one must specify it when the class is being created. This can be accomplished with a metaclass, the type responsible for instantiating types. The metaclass drives the creation of the class object and it can make sure __slots__ makes its way into the class definition dict before the constructor is invoked:

class DefineNew(type):
    def __new__(metacls, name, bases, dct):

        def __new__(cls, *new_args, **new_kwargs):
            return super(defcls, cls).__new__(cls, *new_args, **new_kwargs)

        dct['__slots__'] = ()
        dct['__new__'] = __new__

        defcls = super().__new__(metacls, name, bases, dct)
        return defcls

class C(namedtuple('Foo', "a b c"), metaclass=DefineNew):
    def foo(self):
        return "foo"

Testing results in the expected:

>>> c = C(1, 2, 3)
>>> c.foo()
'foo'
>>> c.bar = 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'C' object has no attribute 'bar'

Metaclass mixing pitfall

Note that the C type object will itself be an instance of DefineMeta - which is not surprising, since that follows from the definition of a metaclass. But this might cause an error if you ever inherit from both C and a type that specifies a metaclass other than type or DefineMeta. Since we only need the metaclass to hook into class creation, but are not using it later, it is not strictly necessary for C to be created as an instance of DefineMeta - we can instead make it an instance of type, just like any other class. This is achieved by changing the line:

        defcls = super().__new__(metacls, name, bases, dct)

to:

        defcls = type.__new__(type, name, bases, dct)

The injection of __new__ and __slots__ will remain, but C will be a most ordinary type with the default metaclass.

In conclusion...

Defining a __new__ which simply calls the superclass __new__ is always superfluous - presumably the real code will also do something different in the injected __new__, e.g. provide the default values for the namedtuple.

---

¹ In the actual class definition the compiler adds a couple of additional items to the class dict, such as the module name. Those are useful, but they do not affect the class definition in the fundamental way that __slots__ does. If X had methods, their function objects would also be included in the dict keyed by function name - automatically inserted as a side effect of executing the def statement in the class definition namespace dict.