How would I create and then import a package containing an unspecified set of modules?

Question

Using Python 3.3 or better, how would I create and then import a package containing an unspecified set of modules? I would like all of the modules in the package to be imported without having to import each of the modules by name. This would allow a new module to be added to a package and the next time the package was imported the new module could be automatically loaded without having to make any other changes to the package.

From outside of the package, I would like importing and using the package to look something like this:

import package
package.module.function()

And from inside of the package, I would like importing and using sibling modules to look something like this:

from . import sibling
sibling.function()

Or this:

import package.sibling
package.sibling.function()

Answer 1

In Python 3.3 or better, search the package directory using pkgutil and load each module using importlib.

In the package's __init__.py:

# Import standard modules.
import importlib
import os
import pkgutil

# For each module in the current directory...
for void, module_name, void in pkgutil.iter_modules([os.path.dirname(__file__)]):

    # Import the module.
    importlib.import_module(__name__+'.'+module_name)

The above can be moved to a library after which the package's __init__.py becomes simply:

# Import local modules.
import loader

# Load all modules in the current directory.
loader.load_all_modules(__file__,__name__)

Note that the importlib machinery was introduced in Python 3.1 and significantly updated in Python 3.3, and the above method has not been tested in Python 3.1.x or 3.2.x.

https://docs.python.org/3.3/library/importlib.html#importlib.import_module https://docs.python.org/3.3/library/pkgutil.html#pkgutil.iter_modules

Miscellaneous Failed Methods

Attempt 1

From outside the package, using absolute import:

import package.module

Requires the module names to be known.

Attempt 2

From the package's __init__.py, using absolute import:

import module

Requires the module names to be known.

Attempt 3

From the package's __init__.py, setting __all__:

__all__ = ['module']

Requires the use of:

from package import *

Which pollutes the namespace and can be dangerous in any case but is especially dangerous in this case because the names of the modules are not known.

Attempt 4

From the package's __init__.py, using pkgutil to identify and import all of the modules in the package appears to do the import but only emulates the native import machinery and does not update the sys.modules dict. This causes issues when modules inside a package try to import sibling modules, for example:

Not being listed in sys.modules causes relative imports to fail with the message:

Parent module '' not loaded, cannot perform relative import.

http://hg.python.org/cpython/file/f7992397e98d/Lib/importlib/_bootstrap.py#l1518

Also, absolute imports reload the module being imported.

https://docs.python.org/3.3/library/imp.html#imp.load_module http://python-notes.curiousefficiency.org/en/latest/python_concepts/import_traps.html#the-double-import-trap

Attempt 5

From the package's __init__.py, using importlib there seems to be no easy way to identify importable modules and sub-packages, except possibly by importing everything and catching the errors, and these false positive error messages make it harder to catch legitimate issues when modules fail to import.