Search code examples
pythonpython-3.xnamespacessetuptoolsdistutils

Why does setup.py sweeps the content of the namespace before installing?

I'm using namespaces with setuptools to distribute a same module in two different repositories. The goal is to get mymodule.one and mymodule.two installed, knowing that the content of one and two comes from different repos. But it looks like two setup.py sweep each other content.

├── repo1
│   ├── mymodule
│   │   ├── __init__.py
│   │   └── one
│   │       └── __init__.py
│   └── setup.py
└── repo2
    ├── mymodule
    │   ├── __init__.py
    │   └── two
    │       └── __init__.py
    └── setup.py

The namespace has the __init__.py below:

test$ cat repo1/mymodule/__init__.py 
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

test$ cat repo2/mymodule/__init__.py 
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)

The setup.py declares the same name:

test$ cat repo1/setup.py 
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='mymodule', packages=find_packages())

test$ cat repo2/setup.py 
#!/usr/bin/env python
from setuptools import setup, find_packages
setup(name='mymodule', packages=find_packages())

Installing the module from the first package allows to import it successfully:

test/repo1$  sudo python3 setup.py install
running install
Checking .pth file support in /usr/local/lib/python3.4/dist-packages/
/usr/bin/python3 -E -c pass
TEST PASSED: /usr/local/lib/python3.4/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
writing dependency_links to mymodule.egg-info/dependency_links.txt
writing mymodule.egg-info/PKG-INFO
writing top-level names to mymodule.egg-info/top_level.txt
reading manifest file 'mymodule.egg-info/SOURCES.txt'
writing manifest file 'mymodule.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/mymodule
copying build/lib/mymodule/__init__.py -> build/bdist.linux-x86_64/egg/mymodule
creating build/bdist.linux-x86_64/egg/mymodule/one
copying build/lib/mymodule/one/__init__.py -> build/bdist.linux-x86_64/egg/mymodule/one
byte-compiling build/bdist.linux-x86_64/egg/mymodule/__init__.py to __init__.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/mymodule/one/__init__.py to __init__.cpython-34.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
mymodule.__pycache__.__init__.cpython-34: module references __path__
creating 'dist/mymodule-0.0.0-py3.4.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing mymodule-0.0.0-py3.4.egg
creating /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Extracting mymodule-0.0.0-py3.4.egg to /usr/local/lib/python3.4/dist-packages
Adding mymodule 0.0.0 to easy-install.pth file

Installed /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Processing dependencies for mymodule==0.0.0
Finished processing dependencies for mymodule==0.0.0

Here's the import:

test/$ ipython3
Python 3.4.3 (default, Oct 14 2015, 20:28:29) [...]

In [1]: from mymodule import [TAB]
extend_path  one

Now we install the other part of the namespace from the second repository:

test/repo2$ sudo python3 setup.py install
running install
Checking .pth file support in /usr/local/lib/python3.4/dist-packages/
/usr/bin/python3 -E -c pass
TEST PASSED: /usr/local/lib/python3.4/dist-packages/ appears to support .pth files
running bdist_egg
running egg_info
creating mymodule.egg-info
writing mymodule.egg-info/PKG-INFO
writing top-level names to mymodule.egg-info/top_level.txt
writing dependency_links to mymodule.egg-info/dependency_links.txt
writing manifest file 'mymodule.egg-info/SOURCES.txt'
reading manifest file 'mymodule.egg-info/SOURCES.txt'
writing manifest file 'mymodule.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/mymodule
copying mymodule/__init__.py -> build/lib/mymodule
creating build/lib/mymodule/two
copying mymodule/two/__init__.py -> build/lib/mymodule/two
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/mymodule
copying build/lib/mymodule/__init__.py -> build/bdist.linux-x86_64/egg/mymodule
creating build/bdist.linux-x86_64/egg/mymodule/two
copying build/lib/mymodule/two/__init__.py -> build/bdist.linux-x86_64/egg/mymodule/two
byte-compiling build/bdist.linux-x86_64/egg/mymodule/__init__.py to __init__.cpython-34.pyc
byte-compiling build/bdist.linux-x86_64/egg/mymodule/two/__init__.py to __init__.cpython-34.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying mymodule.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
mymodule.__pycache__.__init__.cpython-34: module references __path__
creating dist
creating 'dist/mymodule-0.0.0-py3.4.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing mymodule-0.0.0-py3.4.egg
removing '/usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg' (and everything under it)
creating /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Extracting mymodule-0.0.0-py3.4.egg to /usr/local/lib/python3.4/dist-packages
mymodule 0.0.0 is already the active version in easy-install.pth

Installed /usr/local/lib/python3.4/dist-packages/mymodule-0.0.0-py3.4.egg
Processing dependencies for mymodule==0.0.0
Finished processing dependencies for mymodule==0.0.0

But trying to import one again fails since two has trashed it: 

test/$ ipython3
Python 3.4.3 (default, Oct 14 2015, 20:28:29) [...]

In [1]: from mymodule import [TAB]
extend_path  two   

In [1]: from mymodule import one
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-ddf1c613e57c> in <module>()
----> 1 from mymodule import one

ImportError: cannot import name 'one'
Solution

  • There are two requirements for using name spaces correctly.

    1. __init__.py of modules declare a name space
    2. setup.py defines unique name for each module

    The contents of both __init__.py files should be:

    __import__('pkg_resources').declare_namespace(__name__)

    Then setup.py for first module:

    setup(name='mymodule_one', packages=find_packages('.'), 
      namespace_packages=['mymodule'])

    and second module

    setup(name='mymodule_two', packages=find_packages('.'),
      namespace_packages=['mymodule'])

    As a result, should be able to install and import both mymodule.one and mymodule.two

    The common name space mymodule allows both modules to be imported using the same name.

    The name given to setup.py for each module needs to be unique as that is used for the installation path of the module and will overwrite anything that shares it, as you have seen.