I am trying to generate a package from a Python3.6 application using setuptools. While the packaging terminates without errors, the command line program generated by setuptools fail to import modules in the package. Following is the directory tree of my project.
.
├── MANIFEST.in
├── README.md
├── README.rst
├── contributors.txt
├── setup.cfg
├── setup.py
├── sonicparanoid
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-36.pyc
│ │ ├── inpyranoid.cpython-36.pyc
│ │ ├── length_difference_filter.cpython-36.pyc
│ │ ├── ortholog_detection.cpython-36.pyc
│ │ ├── reads_stats.cpython-36.pyc
│ │ ├── seq_tools.cpython-36.pyc
│ │ ├── sonicparanoid.cpython-36.pyc
│ │ ├── sys_tools.cpython-36.pyc
│ │ └── workers.cpython-36.pyc
│ ├── bin
│ │ └── mmseqs
│ ├── blast_tools.py
│ ├── compile_inpyranoid_c.py
│ ├── compile_mmseqs_parser_c.py
│ ├── config.json
│ ├── example
│ │ ├── test_input
│ │ │ ├── chlamydia_trachomatis
│ │ │ ├── deinococcus_radiodurans
│ │ │ ├── gloeobacter_violaceus
│ │ │ └── thermotoga_maritima
│ │ └── test_output
│ │ └── README.txt
│ ├── inpyranoid.py
│ ├── inpyranoid_c.c
│ ├── inpyranoid_c.cpython-36m-darwin.so
│ ├── inpyranoid_c.pyx
│ ├── length_difference_filter.py
│ ├── mmseqs2_src
│ │ ├── README.txt
│ │ └── mmseqs.tar.gz
│ ├── mmseqs_parser_c.c
│ ├── mmseqs_parser_c.cpython-36m-darwin.so
│ ├── mmseqs_parser_c.pyx
│ ├── mmseqs_parser_cython.py
│ ├── ortholog_detection.py
│ ├── quick_multi_paranoid
│ │ ├── Makefile
│ │ ├── Makefile.in
│ │ ├── config
│ │ ├── dump.cpp
│ │ ├── gen_header.cpp
│ │ ├── hashtable.c
│ │ ├── hashtable.h
│ │ ├── hashtable.o
│ │ ├── hashtable_itr.c
│ │ ├── hashtable_itr.h
│ │ ├── hashtable_private.h
│ │ ├── ortholog.c
│ │ ├── qa.h
│ │ ├── qa1
│ │ ├── qa1.cpp
│ │ ├── qa2
│ │ ├── qa2.cpp
│ │ ├── qp
│ │ ├── qp.c
│ │ └── qps.c
│ ├── reads_stats.py
│ ├── seq_tools.py
│ ├── setup_sonicparanoid.py
│ ├── sonicparanoid.py
│ ├── sys_tools.py
│ ├── test_blast_tools.py
│ ├── test_length_difference_filter.py
│ ├── test_ortholog_detection.py
│ ├── test_reads_stats.py
│ ├── test_seq_tools.py
│ ├── test_sonicparanoid.py
│ ├── test_sys_tools.py
│ └── workers.py
└── user_manual.pdf
If I run python3 sonicparanoid.py it works with no problem, but if I create the distribution using setuptools I will get an import errors whenever the main program (sonicparanoid.py) tries to import any other .py module inside the package (sonicparanoid) directory.
Following is the my setup.py:
enter code here
from setuptools import setup, find_packages
from setuptools.extension import Extension
from Cython.Build import cythonize
import numpy
extensions = [
Extension(
"sonicparanoid.inpyranoid_c",
["sonicparanoid/inpyranoid_c.pyx"],
include_dirs=[numpy.get_include()],
),
Extension(
"sonicparanoid.mmseqs_parser_c",
["sonicparanoid/mmseqs_parser_c.pyx"],
include_dirs=[numpy.get_include()],
),
]
from codecs import open
from os import path
here = path.abspath(path.dirname(__file__))
# Get the long description from the README file
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
long_description = f.read()
setup(
name='sonicparanoid',
version='0.0.2', # Required
description='Whatever',
long_description=long_description, # Optional
url='http://iwasakilab.bs.s.u-tokyo.ac.jp/sonicparanoid/', # Optional
author='Me',
author_email='[email protected]',
classifiers=[ # Optional
'Development Status :: 5 - Production/Stable',
'Environment :: Console',
'Intended Audience :: Science/Research',
'Topic :: Scientific/Engineering :: Bio-Informatics',
'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
'Programming Language :: Python :: 3.4',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
],
packages = ['sonicparanoid',], # Required
install_requires=['numpy>=1.14.0', 'pandas>=0.22.0', 'cython>=0.27.0', 'sh>=1.12.14', 'setuptools>=24.2.0'], # specify minimum version
python_requires='>=3.5, <3.7',
ext_modules = cythonize(extensions, compiler_directives={'language_level': 3}),
package_dir={'sonicparanoid': 'sonicparanoid/'},
include_package_data=True,
package_data={'sonicparanoid': ['example/test_output/*', 'example/test_input/*', 'mmseqs2_src/*', 'quick_multi_paranoid/*']
},
entry_points={ # Optional
'console_scripts': [
'sonicparanoid = sonicparanoid.sonicparanoid:main',
],
},
)
This how the import statements look in sonicparanoid.py:
import os
import sys
import platform
import seq_tools as seqtools
at present everything works fine if I execute python3 sonicparanoid.py, but when I use the program generated using setuptools I get the following error:
Traceback (most recent call last):
File "/usr/local/bin/sonicparanoid", line 11, in <module>
load_entry_point('sonicparanoid==0.0.2', 'console_scripts', 'sonicparanoid')()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 572, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2755, in load_entry_point
return ep.load()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2408, in load
return self.resolve()
File "/usr/local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2414, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/usr/local/lib/python3.6/site-packages/sonicparanoid-0.0.2-py3.6- macosx-10.13-x86_64.egg/sonicparanoid/sonicparanoid.py", line 5, in <module>
import seq_tools as seqtools
ModuleNotFoundError: No module named 'seq_tools'
I have tried to use the solution proposed in this question,
but if I change the import with, for example from sonicparanoid import seq_tools as seqtools and run python3 sonicparanoid.py I get an import error ImportError: cannot import name 'seq_tools'
Any help would be greatly appreciated, it is my first attempt in packaging and it is quite frustrating.
Python uses sys.path to find modules/packages and prepends the directory of a script to the beginning of sys.path. With this in mind let's me look into details of import.
When you run python sonicparanoid/sonicparanoid.py Python adds the directory sonicparanoid/ to sys.path. Now when the directory is in sys.path the script can import seq_tools directly because the module seq_tools.py is in a directory in sys.path.
When you install the package and run generated entry point sonicparanoid the directory sonicparanoid/ is not in sys.path (but its parent is) and Python cannot import seq_tools. You have to import it as sonicparanoid.seq_tools. But that means that you cannot import it from sonicparanoid.py when you run sonicparanoid.py as a script!
Bottom line: don't run sonicparanoid.py as a script because sys.path is too different from running entry point importing sonicparanoid package.
Also you script must not be named the same as a Python package. When you have a package sonicparanoid and a script sonicparanoid.py and the script tries to import sonicparanoid — Python tries to import from the script (because it's the first in sys.path) instead of the package and failed.
You can name your script sonicparanoid (without .py extension Python will not try to import from it) or sonic_paranoid.py but not sonicparanoid.py.