How can I redirect module imports with modern Python?

Question

I am maintaining a python package in which I did some restructuring. Now, I want to support clients who still do from my_package.old_subpackage.foo import Foo instead of the new from my_package.new_subpackage.foo import Foo, without explicitly reintroducing many files that do the forwarding. (old_subpackage still exists, but no longer contains foo.py.)

I have learned that there are "loaders" and "finders", and my impression was that I should implement a loader for my purpose, but I only managed to implement a finder so far:

RENAMED_PACKAGES = {
    'my_package.old_subpackage.foo': 'my_package.new_subpackage.foo',
}

# TODO: ideally, we would not just implement a "finder", but also a "loader"
# (using the importlib.util.module_for_loader decorator); this would enable us
# to get module contents that also pass identity checks
class RenamedFinder:

    @classmethod
    def find_spec(cls, fullname, path, target=None):
        renamed = RENAMED_PACKAGES.get(fullname)
        if renamed is not None:
            sys.stderr.write(
                f'WARNING: {fullname} was renamed to {renamed}; please adapt import accordingly!\n')
            return importlib.util.find_spec(renamed)
        return None

sys.meta_path.append(RenamedFinder())

https://docs.python.org/3.5/library/importlib.html#importlib.util.module_for_loader and related functionality, however, seem to be deprecated. I know it's not a very pythonic thing I am trying to achieve, but I would be glad to learn that it's achievable.

Answer 1

On import of your package's __init__.py, you can place whatever objects you want into sys.modules, the values you put in there will be returned by import statements:

from . import new_package
from .new_package import module1, module2
import sys

sys.modules["my_lib.old_package"] = new_package
sys.modules["my_lib.old_package.module1"] = module1
sys.modules["my_lib.old_package.module2"] = module2

If someone now uses import my_lib.old_package or import my_lib.old_package.module1 they will obtain a reference to my_lib.new_package.module1. Since the import machinery already finds the keys in the sys.modules dictionary, it never even begins looking for the old files.

If you want to avoid importing all the submodules immediately, you can emulate a bit of lazy loading by placing a module with a __getattr__ in sys.modules:

from types import ModuleType
import importlib
import sys

class LazyModule(ModuleType):
 def __init__(self, name, mod_name):
  super().__init__(name)
  self.__mod_name = name

 def __getattr__(self, attr):
  if "_lazy_module" not in self.__dict__:
    self._lazy_module = importlib.import(self.__mod_name, package="my_lib")
  return self._lazy_module.__getattr__(attr)

sys.modules["my_lib.old_package"] = LazyModule("my_lib.old_package", "my_lib.new_package")