Search code examples
pythonpython-importlib

`importlib.reload` does not replace the objects in the module's `__dict__`

Running Python 3.7.9 on the python3.7-buster docker container.

Here's a basic example of my issue

First, I import the json module and verify that it contains the decoder module as an attribute:

>>> import json, importlib
>>> json.decoder
<module 'json.decoder' from '/usr/local/lib/python3.7/json/decoder.py'>

Then, I set that attribute to None and verify:

>>> json.__dict__['decoder'] = None
>>> json.decoder     # None's repr is blank

Then I attempt to reload the module and re-check the status of json.decoder. I expect it will be a module again, but what I find is:

>>> importlib.reload(json)
<module 'json' from '/usr/local/lib/python3.7/json/__init__.py'>
>>> json.decoder    # None's repr is blank

OK, am I able to import the decoder directly? No, it simply returns the value from the module that is already loaded. (This is not totally unexpected.)

>>> from json import decoder as new_decoder
>>> new_decoder     # None's repr is blank

OK, I notice that importlib.reload() returns a module. What if I set the module identifier equal to the function call?

>>> json = importlib.reload(json)
>>> json.decoder    # None's repr is blank

Here's what seems like a relevant snippet from the Python documentation on reload:

When reload() is executed:

Python module’s code is recompiled and the module-level code re-executed, defining a new set of objects which are bound to names in the module’s dictionary by reusing the loader which originally loaded the module. The init function of extension modules is not called a second time. ... The names in the module namespace are updated to point to any new or changed objects

This seems to precisely describe the behavior I expected, but not what I've encountered. What's going on here?

Solution

  • Reexecuting the code of the json package just doesn't reassign json.decoder. There's nothing in the json package's __init__.py that would do that.

    The json package's __init__.py contains the following line:

    from .decoder import JSONDecoder, JSONDecodeError

    The first time this line is run (on the first import of json), the json.decoder submodule is initialized, and as part of that initialization, a reference to that submodule is set as the decoder attribute of the json module object.

    When you reload json, this line is reexecuted, but the json.decoder submodule has already been initialized and does not need to be initialized again. json.decoder is not reassigned.

    In general, you should not expect reloading a module to put it in any kind of sane or useful state. Module reloading has tons of weird and subtle issues, such as creating "evil twin" classes, not updating anything imported from the module with from, not playing well with extension modules, etc.