Search code examples
pythonpython-3.xpython-importsetup.pypyc

ModuleNotFoundError with setup.py using a compiled pyc module


I can normally import a compiled .pyc module as well as .py, but when trying to package a simple project with setup.py, I'm getting the ModuleNotFoundError exception for the compiled .pyc module. Because this is only happening when using setup.py, otherwise is working fine, I don't know if there's something I should had to setup.py to make this work.

The project structure is currently something like this:

proj
├── FAILING.pyc
├── __init__.py
├── aux/
│   ├── __init__.py
│   └── aux.c
└── main.py

and the setup.py:

from setuptools import setup, Extension, find_packages

DISTNAME = 'proj'

INSTALL_REQUIRES = [
        'cython>=0.29.13',
        'numpy>=1.16.4'
]
PYTHON_REQUIRES = '>=3.6'

ENTRY_POINTS = {
    'console_scripts': ['proj = proj.main:main']
}

def setup_extensions(metadata):
    ext_modules = [Extension('proj.aux.aux', sources=['proj/aux/aux.c'])]
    metadata['ext_modules'] = ext_modules

def setup_package():
    metadata = dict(
        name=DISTNAME,
        version='0.1',
        package_dir={'': '.'},
        packages=find_packages(),
        entry_points=ENTRY_POINTS,
        python_requires=PYTHON_REQUIRES,
        install_requires=INSTALL_REQUIRES,
        zip_safe=False,
    )
    setup_extensions(metadata)
    setup(**metadata)


if __name__ == '__main__':
    setup_package()

The main.py:

#!/usr/bin/env python3

import proj.aux.aux as aux
import proj.FAILING

def main():
    print('Hello World')

If I just try to import FAILING.pyc on the repl everything works as expected:

>>> import FAILING
>>>

But if I first run python3 setup.py intall and then call proj I'm getting the following error:

$ proj
Traceback (most recent call last):
  File "/path/to/bin/proj", line 11, in <module>
    load_entry_point('proj==0.1', 'console_scripts', 'proj')()
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
    return ep.load()
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/path/to/lib/python3.8/site-packages/proj-0.1-py3.8-macosx-10.14-x86_64.egg/proj/main.py", line 4, in <module>
    import proj.FAILING
ModuleNotFoundError: No module named 'proj.FAILING'

I'm also running this inside a virtualenv environment, although I'm guessing this is not related to the error.

What am I doing wrong, or what would I need to change to make this work?


Solution

  • Here is a small demo which builds a source distribution and wheel that contains a .pyc file

    note that I've removed most of the cruft from your example as the cython stuff is unrelated to your problem

    set -euxo pipefail
    
    rm -rf dist testpkg setup.py
    
    cat > setup.py <<EOF
    from setuptools import setup
    
    setup(
        name='foo',
        version='1',
        packages=['testpkg'],
        package_data={'testpkg': ['*.pyc']},
    )
    EOF
    
    mkdir testpkg
    touch testpkg/__init__.py
    echo 'print("hello hello world")' > testpkg/mod.py
    
    python3 -m compileall -b testpkg/mod.py
    rm testpkg/mod.py
    
    python3 setup.py sdist bdist_wheel
    tar --list -f dist/*.tar.gz
    unzip -l dist/*.whl
    

    there's a few things to note about the setup.py:

    • I include in packages the package that has the .pyc files -- I could have used setuptools.find_packages instead, but this was simpler
    • The .pyc file is included as package_data -- by default pyc files are not packaged as they're generally leftover build artifacts
    • I need to compile the pyc into the legacy location using the -b flag of python3 -m compileall
    • even a "compiled" pyc file does not obfuscate the actual code, it can be recovered using dis for example -- when you talk about "compiled" here it just means it has been transformed into the (still relatively high level) python bytecode

    From either the source distribution or the wheel, you can install the package.

    For example running the script:

    $ bash t.sh
    + rm -rf dist testpkg setup.py
    + cat
    + mkdir testpkg
    + touch testpkg/__init__.py
    + echo 'print("hello hello world")'
    + python3 -m compileall -b testpkg/mod.py
    Compiling 'testpkg/mod.py'...
    + rm testpkg/mod.py
    + python3 setup.py sdist bdist_wheel
    running sdist
    running egg_info
    writing foo.egg-info/PKG-INFO
    writing dependency_links to foo.egg-info/dependency_links.txt
    writing top-level names to foo.egg-info/top_level.txt
    reading manifest file 'foo.egg-info/SOURCES.txt'
    writing manifest file 'foo.egg-info/SOURCES.txt'
    warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md
    
    running check
    warning: check: missing required meta-data: url
    
    warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
    
    creating foo-1
    creating foo-1/foo.egg-info
    creating foo-1/testpkg
    copying files to foo-1...
    copying setup.py -> foo-1
    copying foo.egg-info/PKG-INFO -> foo-1/foo.egg-info
    copying foo.egg-info/SOURCES.txt -> foo-1/foo.egg-info
    copying foo.egg-info/dependency_links.txt -> foo-1/foo.egg-info
    copying foo.egg-info/top_level.txt -> foo-1/foo.egg-info
    copying testpkg/__init__.py -> foo-1/testpkg
    copying testpkg/mod.pyc -> foo-1/testpkg
    Writing foo-1/setup.cfg
    creating dist
    Creating tar archive
    removing 'foo-1' (and everything under it)
    running bdist_wheel
    running build
    running build_py
    copying testpkg/__init__.py -> build/lib/testpkg
    copying testpkg/mod.pyc -> build/lib/testpkg
    installing to build/bdist.linux-x86_64/wheel
    running install
    running install_lib
    creating build/bdist.linux-x86_64/wheel
    creating build/bdist.linux-x86_64/wheel/testpkg
    copying build/lib/testpkg/__init__.py -> build/bdist.linux-x86_64/wheel/testpkg
    copying build/lib/testpkg/mod.pyc -> build/bdist.linux-x86_64/wheel/testpkg
    running install_egg_info
    Copying foo.egg-info to build/bdist.linux-x86_64/wheel/foo-1-py3.8.egg-info
    running install_scripts
    creating build/bdist.linux-x86_64/wheel/foo-1.dist-info/WHEEL
    creating 'dist/foo-1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
    adding 'testpkg/__init__.py'
    adding 'testpkg/mod.pyc'
    adding 'foo-1.dist-info/METADATA'
    adding 'foo-1.dist-info/WHEEL'
    adding 'foo-1.dist-info/top_level.txt'
    adding 'foo-1.dist-info/RECORD'
    removing build/bdist.linux-x86_64/wheel
    + tar --list -f dist/foo-1.tar.gz
    foo-1/
    foo-1/PKG-INFO
    foo-1/foo.egg-info/
    foo-1/foo.egg-info/PKG-INFO
    foo-1/foo.egg-info/SOURCES.txt
    foo-1/foo.egg-info/dependency_links.txt
    foo-1/foo.egg-info/top_level.txt
    foo-1/setup.cfg
    foo-1/setup.py
    foo-1/testpkg/
    foo-1/testpkg/__init__.py
    foo-1/testpkg/mod.pyc
    + unzip -l dist/foo-1-py3-none-any.whl
    Archive:  dist/foo-1-py3-none-any.whl
      Length      Date    Time    Name
    ---------  ---------- -----   ----
            0  2021-02-17 22:27   testpkg/__init__.py
          136  2021-02-17 22:27   testpkg/mod.pyc
          163  2021-02-17 22:28   foo-1.dist-info/METADATA
           92  2021-02-17 22:28   foo-1.dist-info/WHEEL
            8  2021-02-17 22:27   foo-1.dist-info/top_level.txt
          408  2021-02-17 22:28   foo-1.dist-info/RECORD
    ---------                     -------
          807                     6 files
    

    Afterwards, I can install this package and use it:

    $ mkdir t
    $ cd t
    $ virtualenv venv
    ...
    $ . venv/bin/activate
    $ pip install ../dist/foo-1-py3-none-any.whl
    ...
    $ python3 -c 'import testpkg.mod'
    hello hello world