Search code examples
pythonpackagingsetuptools

Should I include tests and .pyc files when building package with setuptools?

I'm trying to install a local package using pip install . in a conda environment. Everything looks okay, except that when I run this command, there are .pyc files being included by default (I'm guessing this is due to the build process). Explicitly excluding them in MANIFEST.in doesn't help. First of all, is this a problem? The setup.py is actually very simple. Mainly, the only relevant thing is include_package_data = True that I need in order to include some static files.

setup.py

from setuptools import setup, find_packages
setup(
...
    include_package_data = True,
...
)

and this is the MANIFEST.in

MANIFEST.in

recursive-include <my-project>/static *
global-exclude *.pyc
global-exclude .git

Another question: Should I include my test folder (recursive-include <my-project>/test/ *) in the installed package? I have some tests using nose and other client-side tests which are included as static files. What is the recommended way to deal with tests in this case?

Solution

  • The discussion is getting long, so I'll just write up my thoughts here.

    Everything looks okay, except that when I run this command, there are .pyc files being included by default (I'm guessing this is due to the build process).

    Python sources are precompiled on installation, and .pyc files are put along with the .py files in the installation location by setuptools. It's a normal behavior.

    Explicitly excluding them in MANIFEST.in doesn't help.

    MANIFEST.in governs distribution, not installation. That is, it tells setuptools which files to include in the archive when python setup.py sdist is run. It is commonly used to include documentation, tests, license and other accompanying files that are part of the package, but are not actually installed. The files that are installed (in addition to .py files) are specified in the package_data keyword argument to setup().

    Another question: Should I include my test folder (recursive-include /test/ *) in the installed package?

    There are two approaches possible:

    1. Include tests in the distribution, but not install them. To do this, you put them in a separate folder and include all required files (including the .py ones) explicitly in the MANIFEST.in. As an example, see this excerpt from the manifest of one of my projects:

      recursive-include test *.py
include test/pylintrc
include test/.coveragerc

    2. Include tests in the package itself, so that you can run them as import package; package.test() (for instance, numpy does that). No need to tinker with the MANIFEST.in, just put them in the main package directory (as a subpackage or whatever) and treat them as a regular subpackage/module.

    Edit: answering the additional questions:

    What about the option test_suite (e.g. nose.readthedocs.org/en/latest/setuptools_integration.html)?

    I haven't used this particular option (maybe I should), but I have a similar setup in another project, which also makes python setup.py test available — used by Travis.CI for automatic test runs. Since I'm a fan of py.test, it requires some interfacing; for nose it would look simpler.

    In general, if you can do it, do it, it's a good thing to have (for continuous integration in particular).

    Does including tests as a subpackage cause issues?

    Well, numpy and many other packages are doing fine. The only downsides that I see are:

    • the size of the installation is increased (but in our age of terabyte HDDs it is probably not that important)
    • tests are somewhat harder to run from the command line
    • you don't have a clear separation between the package itself and the tests (very arguable)

    In conclusion, I don't have anything against it and may very well use this kind of setup for my next project.

    The Python Packaging User Guide includes an example that does this: packages=find_packages(exclude=['docs', 'tests*'])

    This is done to include all Python packages for installation, but skip the folders with .py files that are not supposed to be installed as packages. You can either do that, or supply the list of packages manually. It's just an example of how to implement the scenario 1 from my answer above. In the scenario 2 your tests folder will be inside your main package folder, and will be picked up by find_packages() normally.