How to use setup.py and setup.cfg and pip-tools to obtain layered requirements.txt under different environment of a Django project?

I'm using https://github.com/jazzband/pip-tools to handle compiling the requirements.txt file for a Django project.

Previously, I was using it without a setup.py, and so I was using base.in, local.in, and production.in.

When I needed a local requirements.txt, after I finished running pip-compile, I just ran pip-sync base.txt local.txt, and it installed the requirements for the local environment.

When I needed a production requirements.txt, after I finished running pip-compile, I just ran pip-sync base.txt production.txt, and it installed the requirements for the production environment.

So, I switched away from using base.in, because I wanted to also lock the Python version and I realized setup.py and setup.cfg can help using python_requires.

But now I've become unsure of how to use setup.py and setup.cfg along with pip-tools to compile requirements.txt that can be environment-specific.

The only documentation for layered requirements is by using the different .in files as written in the README as in: https://github.com/jazzband/pip-tools#workflow-for-layered-requirements

So, my question is, with the following components:

pip-tools
setup.py and setup.cfg

How to still have layered requirements?

Solution

pip-tools works smoothly with setup.py too. You just need to run it without providing it with an *.in file.

Mixed variant with setup.py and in files:

So assuming, that you have the following structure:

# setup.py replacing the base.in file
from distutils.core import setup


setup(
    name="MyLibrary",
    version="1.0",
    install_requires=[
        "requests",
        "bcrypt",
    ]
)

and a local.in file:

django

you need to do the following to compile the dependencies:

$ pip-compile -o base.txt 
$ pip-compile local.in
$ pip-sync base.txt local.txt

$ pip-compile -o base.txt will generate dependencies from setup.py using base.txt as an output file. It defaults to requirements.txt. $ pip-compile local.in is the same what you did before, as with the pip-sync part which doesn't change.

So the only magic here is to run pip-compile without providing it with an input file.

Setup.py only solution:

Setup.py supports extras_require which is a dictionary of named optional dependencies:

[...]
    extras_require={
        "local": ["pytest"],
    },
[...]

pip-tools has an option extra:

$ pip-compile --help |grep extra
  --extra TEXT     Name of an extras_require group to install;

So you could do the following:

pip-compile --extra local -o local.txt
pip-compile --extra production -o production.txt

The output files contain all requiremenets in install_requires + the extra requirements specified.

Afterwards you just sync the local/production.txt:

$ pip-sync local.txt
$ pip-sync production.txt

If I was you, I would grab the pure setup.py variant.