Deeply nested namespaced packages

I need to maintain some ETL tool constructed in such a way tasks and pipelines are defined as collection of python packages. Think about plugin architecture with small core and almost thousand of plugins in nested namespaces/packages/subpackages. This is not a hot mess yet, overall quality is quite good, but setup.py and __init__.py-s look very hacky and sometimes cause unexpected problems during imports.

I would like to simplify this a bit. Since Python 3.3 we can put packages in namespaces simply by creating subdirectories without __init__.py. This is exactly what I need, but I would like to avoid deeply nested subdirectories in source code, because a large amount of packages is very small. In extreme they would look like this:

$ tree
.
├── setup.cfg
├── setup.py
└── src
    └── foo
        └── bar
            └── baz
                └── xyz
                    └── uvw
                        └── package
                            ├── actual_code.py
                            └── __init__.py

Is there a way to use implicit namespaces without so deep structure and simply specify namespace somewhere in setup.py (or even better setup.cfg)? In other words, is there a simple way to tell: install package in foo.bar.baz.xyz.uvw namespace?

I would like to have structure like this:

$ tree
.
├── setup.cfg
├── setup.py
└── src
    └── package
        ├── actual_code.py
        └── __init__.py

but installation process should put package into foo/bar/baz/xyz/uvw/package folder, so it could be imported with full path.

Edit: Is this even a good idea?

Solution

This is possible using the package_dir argument to distutils.core.setup (or equivalent from setuptools).

Just modify your setup.py to contain something like:

from distutils.core import setup

setup(# ... other setup arguments ...
      package_dir={'foo.bar.baz.xyz.uvw': 'src'},
      packages=['foo.bar.baz.xyz.uvw.package'],
     )

The key part here is that package_dir is saying "the contents of foo.bar.baz.xyz.uvw are what is found in the src directory", while packages=['foo.bar.baz.xyz.uvw.package'] tells it to expect to find and install a package named foo.bar.baz.xyz.uvw.package.

The setup.cfg equivalent would be:

[options]
package_dir=
    foo.bar.baz.xyz.uvw=src
packages = foo.bar.baz.xyz.uvw.package