How to resolve conflicting dependencies in tox?

I have been using tox to run the lintin packes over my code base. However I have ran into the issue of not having the dependecies up to date with my gitlab pipeline, because I was not updating my dependencies not to affect the deployed version. For this reason I wanted to switch to using requirements-dev.txt in my tox command. My current setup is as follow:

[tox]
envlist = py3, flake8, black, mypy, pylint
basepython=python3

[testenv]
deps =
    -rrequirements.txt
    -rrequirements-dev.txt
commands =
    python -m pytest -v --cov={[vars]SOURCE} \
                        --cov-config=tox.ini \
                        --cov-report=html \
                        --cov-fail-under={[vars]MINIMAL_COVERAGE} \
                        --cov-report term-missing \
                        {[vars]TEST_DIR}

[testenv:pylint]
basepython = python3
deps =
    -rrequirements.txt
    -rrequirements-dev.txt
# ignored:
# R0903 - Too few public methods
# W3101 - missing-timeout
commands = python -m pylint {[vars]SOURCE} --rcfile=tox.ini -d R0903,W0511,W3101
# pylint configuration
[pylint.MAIN]
load-plugins=pylint_flask_sqlalchemy,pylint_flask

[pylint]
max-args = 10

[testenv:flake8]
basepython=python3
deps =
    -rrequirements-dev.txt
commands= python3 -m flake8 --max-line-length=100 {[vars]SOURCE} {[vars]TEST_DIR}
whitelist_externals = /usr/bin/python3

# flake8 config
[flake8]
ignore = E722, W503, W504, N818, F841

[testenv:mypy]
deps =
    -rrequirements-dev.txt
commands = mypy --install-types --non-interactive \
                --ignore-missing-imports \
                --disallow-untyped-defs \
                --disallow-incomplete-defs \
                --disallow-untyped-decorators {[vars]SOURCE} {[vars]TEST_DIR}

[testenv:black]
deps = -rrequirements-dev.txt
commands = black --check --diff {[vars]SOURCE} {[vars]TEST_DIR}

# Format code automatically using black rules
[testenv:black-format]
deps = -rrequirements-dev.txt
commands = black {[vars]SOURCE} {[vars]TEST_DIR}

As you can see, i've had to keep the requirements.txt in dependencies for testenv and pylint (I had some dependencies that I needed in both), however I am now getting a double requirement error such as ERROR: Double requirement given: click==8.1.3 (from -r requirements-dev.txt (line 13)) (already in click==8.0.3 (from -r requirements.txt (line 19)), name='click').

I am wondering what is the most elegant solution to this problem? Omitting those dependencies that are covered in requirements.txt in requirements-dev.txt? Or just keeping the version lower?

Thank you for your advice

Solution

The most elegant, and widely used solution is not to run each linter in a separate tox environment, but to have one linter environment, which runs pre-commit.

pre-commit is a linter runner and both takes care of running the linters and dependency management of the linters.

Your tox.ini would look like that:

[testenv:lint]
deps= pre-commit
commands = pre-commit run --all-files

More info on pre-commit https://pre-commit.com/

Example configuration from one of my projects https://github.com/jugmac00/flask-reuploaded/blob/723fe4e355cd260bc82bf4f1c712036ae3d3d4b6/tox.ini#L24

https://github.com/jugmac00/flask-reuploaded/blob/master/.pre-commit-config.yaml

Disclaimer: I am one of the tox maintainers