I can normally import a compiled .pyc
module as well as .py
, but when trying to package a simple project with setup.py
, I'm getting the ModuleNotFoundError
exception for the compiled .pyc
module. Because this is only happening when using setup.py
, otherwise is working fine, I don't know if there's something I should had to setup.py
to make this work.
The project structure is currently something like this:
proj
├── FAILING.pyc
├── __init__.py
├── aux/
│ ├── __init__.py
│ └── aux.c
└── main.py
and the setup.py
:
from setuptools import setup, Extension, find_packages
DISTNAME = 'proj'
INSTALL_REQUIRES = [
'cython>=0.29.13',
'numpy>=1.16.4'
]
PYTHON_REQUIRES = '>=3.6'
ENTRY_POINTS = {
'console_scripts': ['proj = proj.main:main']
}
def setup_extensions(metadata):
ext_modules = [Extension('proj.aux.aux', sources=['proj/aux/aux.c'])]
metadata['ext_modules'] = ext_modules
def setup_package():
metadata = dict(
name=DISTNAME,
version='0.1',
package_dir={'': '.'},
packages=find_packages(),
entry_points=ENTRY_POINTS,
python_requires=PYTHON_REQUIRES,
install_requires=INSTALL_REQUIRES,
zip_safe=False,
)
setup_extensions(metadata)
setup(**metadata)
if __name__ == '__main__':
setup_package()
The main.py
:
#!/usr/bin/env python3
import proj.aux.aux as aux
import proj.FAILING
def main():
print('Hello World')
If I just try to import FAILING.pyc
on the repl everything works as expected:
>>> import FAILING
>>>
But if I first run python3 setup.py intall
and then call proj
I'm getting the following error:
$ proj
Traceback (most recent call last):
File "/path/to/bin/proj", line 11, in <module>
load_entry_point('proj==0.1', 'console_scripts', 'proj')()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/path/to/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/path/to/lib/python3.8/site-packages/proj-0.1-py3.8-macosx-10.14-x86_64.egg/proj/main.py", line 4, in <module>
import proj.FAILING
ModuleNotFoundError: No module named 'proj.FAILING'
I'm also running this inside a virtualenv environment, although I'm guessing this is not related to the error.
What am I doing wrong, or what would I need to change to make this work?
Here is a small demo which builds a source distribution and wheel that contains a .pyc file
note that I've removed most of the cruft from your example as the cython stuff is unrelated to your problem
set -euxo pipefail
rm -rf dist testpkg setup.py
cat > setup.py <<EOF
from setuptools import setup
setup(
name='foo',
version='1',
packages=['testpkg'],
package_data={'testpkg': ['*.pyc']},
)
EOF
mkdir testpkg
touch testpkg/__init__.py
echo 'print("hello hello world")' > testpkg/mod.py
python3 -m compileall -b testpkg/mod.py
rm testpkg/mod.py
python3 setup.py sdist bdist_wheel
tar --list -f dist/*.tar.gz
unzip -l dist/*.whl
there's a few things to note about the setup.py:
packages
the package that has the .pyc
files -- I could have used setuptools.find_packages
instead, but this was simpler.pyc
file is included as package_data
-- by default pyc
files are not packaged as they're generally leftover build artifacts-b
flag of python3 -m compileall
dis
for example -- when you talk about "compiled" here it just means it has been transformed into the (still relatively high level) python bytecodeFrom either the source distribution or the wheel, you can install the package.
For example running the script:
$ bash t.sh
+ rm -rf dist testpkg setup.py
+ cat
+ mkdir testpkg
+ touch testpkg/__init__.py
+ echo 'print("hello hello world")'
+ python3 -m compileall -b testpkg/mod.py
Compiling 'testpkg/mod.py'...
+ rm testpkg/mod.py
+ python3 setup.py sdist bdist_wheel
running sdist
running egg_info
writing foo.egg-info/PKG-INFO
writing dependency_links to foo.egg-info/dependency_links.txt
writing top-level names to foo.egg-info/top_level.txt
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt, README.md
running check
warning: check: missing required meta-data: url
warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied
creating foo-1
creating foo-1/foo.egg-info
creating foo-1/testpkg
copying files to foo-1...
copying setup.py -> foo-1
copying foo.egg-info/PKG-INFO -> foo-1/foo.egg-info
copying foo.egg-info/SOURCES.txt -> foo-1/foo.egg-info
copying foo.egg-info/dependency_links.txt -> foo-1/foo.egg-info
copying foo.egg-info/top_level.txt -> foo-1/foo.egg-info
copying testpkg/__init__.py -> foo-1/testpkg
copying testpkg/mod.pyc -> foo-1/testpkg
Writing foo-1/setup.cfg
creating dist
Creating tar archive
removing 'foo-1' (and everything under it)
running bdist_wheel
running build
running build_py
copying testpkg/__init__.py -> build/lib/testpkg
copying testpkg/mod.pyc -> build/lib/testpkg
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/__init__.py -> build/bdist.linux-x86_64/wheel/testpkg
copying build/lib/testpkg/mod.pyc -> build/bdist.linux-x86_64/wheel/testpkg
running install_egg_info
Copying foo.egg-info to build/bdist.linux-x86_64/wheel/foo-1-py3.8.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/foo-1.dist-info/WHEEL
creating 'dist/foo-1-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'testpkg/__init__.py'
adding 'testpkg/mod.pyc'
adding 'foo-1.dist-info/METADATA'
adding 'foo-1.dist-info/WHEEL'
adding 'foo-1.dist-info/top_level.txt'
adding 'foo-1.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
+ tar --list -f dist/foo-1.tar.gz
foo-1/
foo-1/PKG-INFO
foo-1/foo.egg-info/
foo-1/foo.egg-info/PKG-INFO
foo-1/foo.egg-info/SOURCES.txt
foo-1/foo.egg-info/dependency_links.txt
foo-1/foo.egg-info/top_level.txt
foo-1/setup.cfg
foo-1/setup.py
foo-1/testpkg/
foo-1/testpkg/__init__.py
foo-1/testpkg/mod.pyc
+ unzip -l dist/foo-1-py3-none-any.whl
Archive: dist/foo-1-py3-none-any.whl
Length Date Time Name
--------- ---------- ----- ----
0 2021-02-17 22:27 testpkg/__init__.py
136 2021-02-17 22:27 testpkg/mod.pyc
163 2021-02-17 22:28 foo-1.dist-info/METADATA
92 2021-02-17 22:28 foo-1.dist-info/WHEEL
8 2021-02-17 22:27 foo-1.dist-info/top_level.txt
408 2021-02-17 22:28 foo-1.dist-info/RECORD
--------- -------
807 6 files
Afterwards, I can install this package and use it:
$ mkdir t
$ cd t
$ virtualenv venv
...
$ . venv/bin/activate
$ pip install ../dist/foo-1-py3-none-any.whl
...
$ python3 -c 'import testpkg.mod'
hello hello world