How can I use super() in both elegant and safe (regarding Python module reloading) way?

Question

I discovered an annoying behaviour of reload() (both Python 2 builtin and from importlib) that I am trying to walkarround.

I am analysing data in interactive Python interpreter. My code is organised in modules (both Python 2 and 3 compatible) which I often change.

Restarting the interpreter is not feasible due to long time of loading data, so I prefer to recursively reload modules instead.

The problem is that reload() updates the code but preserves the module global scope (it applies to Python 3 importlib.reload() as well). It seems to be harmful for methods using super() (it took me a while to realise what is going on).

The minimal failing example for a module bar.py:

class Bar(object):
    def __init__(self):
        super(Bar, self).__init__()

is:

>>> import bar
>>> class Foo(bar.Bar):
...     pass
... 
>>> reload(bar)
<module 'bar' from '[censored]/bar.py'>
>>> Foo()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[censored]/bar.py", line 3, in __init__
    super(Bar, self).__init__()
TypeError: super(type, obj): obj must be an instance or subtype of type

I may:

1. use super() without arguments in Python 3 manner (which is not compatible with Python 2),
2. abandon it and call Bar.__init__(self) instead (which is harder to maintain and discouraged),
3. monkey-patch the class adding a class attribute containing a circular reference to the class itself.

None of the ideas I like. Is there any other way of dealing with the issue?

Answer 1

You can't, because it's basically impossible to do anything elegantly and safely where module reloading is concerned. That kind of thing is super tricky to get right.

The specific way that problem is manifesting here is that Foo's superclass is still the old Bar, but the old Bar refers to itself by name, and the Bar name now refers to the new Bar in the namespace where it's looking. The old Bar is trying to pass the new Bar to super.

---

I can give you some options. All of them are inelegant and unsafe and probably going to give you weird surprises eventually, but all that is also true about module reloading.

Probably the most easily-understood option is to rerun the Foo class definition to get a new Foo class descending from the new Bar:

reload(bar)
class Foo(bar.Bar):
    pass

New instances of Foo will then use the new Bar, and will not experience problems with super. Old instances will use the old Foo and the old Bar. They're not going to have problems with __init__, because that already ran, but they're likely to have other problems.

Another option you can take is to update Foo's superclass so Foo now descends from the new Bar:

reload(bar)

Foo.__bases__ = (bar.Bar,)

New and old instances of Foo will then use the new Bar. Depending on what you changed in Bar, the new class may not be compatible with the old instances, especially if you changed what data Bar keeps on its instances or how Bar.__init__ performs initialization.