Search code examples
pythondjangopython-importlib

Django autoreload raises: TypeError: unhashable type: 'types.SimpleNamespace'

When I upgrade importlib_meta from version 8.4.0 to 8.5.0 (released just yesterday, Sep 11 2024), I get the following error when I start running the development server with python manage.py runserver:

   File "/app/manage.py", line 17, in main
     execute_from_command_line(sys.argv)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
     utility.execute()
   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute
     self.fetch_command(subcommand).run_from_argv(self.argv)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 413, in run_from_argv
     self.execute(*args, **cmd_options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 75, in execute
     super().execute(*args, **options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 459, in execute
     output = self.handle(*args, **options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 112, in handle
     self.run(**options)
   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/runserver.py", line 119, in run
     autoreload.run_with_reloader(self.inner_run, **options)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 671, in run_with_reloader
     start_django(reloader, main_func, *args, **kwargs)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 660, in start_django
     reloader.run(django_main_thread)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 344, in run
     self.run_loop()
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 350, in run_loop
     next(ticker)
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 390, in tick
     for filepath, mtime in self.snapshot_files():
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 411, in snapshot_files
     for file in self.watched_files():
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 304, in watched_files
     yield from iter_all_python_module_files()
   File "/usr/local/lib/python3.10/site-packages/django/utils/autoreload.py", line 120, in iter_all_python_module_files
     return iter_modules_and_files(modules, frozenset(_error_files))
 TypeError: unhashable type: 'types.SimpleNamespace'

I actually could narrow the problem down to the following commit https://github.com/python/importlib_metadata/commit/56b61b3dd90df2dba2da445a8386029b54fdebf3.

When I install importlib_meta just one commit before the problematic commit via pip install git+https://github.com/python/importlib_metadata@d968f6270d55f27a10491344a22e9e0fd77b5583 the error disappears. When I install importlib_meta at the problematic commit the error starts to appear.

I can not really make sense out of the Traceback and how the problem might be connected to the changes of the mentioned commit. Has anyone an idea what could cause this problem or how I can debug it?

Update (Sep 15, 2024)

The problem is solved now with version 3.20.2 of the zipp package.

Solution

  • The problematic commit in importlib_meta puts a types.SimpleNamespace object in sys.modules["zipp.compat.overlay.zipfile"]:

    zipfile = types.SimpleNamespace(**vars(importlib.import_module('zipfile')))
...
sys.modules[__name__ + '.zipfile'] = zipfile  # type: ignore[assignment]

    Here's the invocation error site in django:

        modules = tuple(
        m
        for m in map(sys.modules.__getitem__, keys)
        if not isinstance(m, weakref.ProxyTypes)
    )
    # WARNING: `modules` now has `zipfile = types.SimpleNamespace(...)` in it
    return iter_modules_and_files(modules, frozenset(_error_files))


@lru_cache(maxsize=1)
def iter_modules_and_files(modules, extra_files):

    From the Python docs on @functools.lru_cache:

    Since a dictionary is used to cache results, the positional and keyword arguments to the function must be hashable.

    types.SimpleNamespace() is not hashable:

    >>> from types import SimpleNamespace
>>> hash(SimpleNamespace())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'types.SimpleNamespace'

    I would say that this is a django problem - django should account for the fact that sys.modules is subject to monkeypatching at runtime, and can hold arbitrary objects (including unhashable ones).