Poetry + Tox + Pytest seems to ALWAYS run every test

I have an issue with poetry + tox + pytest where if I run poetry run tox tests or poetry run tox py39-test every test env runs. Additionally, if I use py39-test, then it seems to skip the pytest commands all together. Using tox <command> from a poetry shell doesn't change how it outputs.

I am going to try and post specific code on my issue. However, code snippets and project can be found at https://github.com/flamusdiu/xleapp.

First, log output from tox tests (gist) shows that I ran the tool but both the tests and the lint env ran.

[tox]
isolated_build = true
envlist = tests, lint 

[testenv:tests]
commands =
  pytest -vv --cov={envsitepackagesdir}/xleapp --cov-report xml --cov-report html --ignore=data
deps =
  requests
  tqdm
extras =
  tests

[testenv:lint]
commands =
   black -S src/
   flakeheaven lint src/
extras = 
   lint

If you change the envlist to {py39, py310}-{tests, lint}, then you get this tox error from tox py39-tests (gist) which shows each of the tests runs but run-test never runs. It seems to skip any commands.This should run the commands for each tests.

Also, when running a specific python version (py39-) should run only that test env.

I have looked everywhere trying to figure out what is going on. Maybe its a bug? Maybe it's something else?

References:

The references are just some of the background on what I have setup. I am using the pyproject.toml file and copying to the tox.ini file has the same problem.

Update 1

Changed the tox configuration to:

[tox]
isolated_build = true

[testenv]
whitelist_externals = poetry
commands =
  poetry install -v
  poetry run pytest tests/
extras=
   tests

[testenv:lint]
commands =
   black -S src/
   flakeheaven lint src/
extras = 
   lint

This works. @jürgen-gmach pointed out the slight issue with my current build which was driving me bonkers.

Solution

There are a couple of issues...

a) remove this from your pyproject.toml as you want to create a tox.ini anyway - and that is preferable

[tool.tox]
legacy_tox_ini = """
[tox]
isolated_build = true
envlist = {py39, py310}-{tests, lint}

[testenv:tests]
commands =
  pytest -vv --cov={envsitepackagesdir}/xleapp --cov-report xml --cov-report html --ignore=data
deps =
  requests
  tqdm
extras =
  tests

[testenv:lint]
commands =
   black -S src/
   flakeheaven lint src/
extras = 
   lint
  """

b) your tox.ini is almost ok

Usually, you have one base testenv for all tests, and without a dedicated name. Then you can run tests for e.g. Python 3.9 with tox -e py39. And your linter with tox -e lint, or all with tox.

[tox]
isolated_build = true
envlist = py39, lint 

[testenv]
commands =
  pytest -vv --cov={envsitepackagesdir}/xleapp --cov-report xml --cov-report html --ignore=data
deps =
  requests
  tqdm
extras =
  tests

[testenv:lint]
commands =
   black -S src/
   flakeheaven lint src/
extras = 
   lint

c) The deps requests and tqdm is a code smell - these are test dependencies, so they should be installed via the extras = tests not separately.

d) you should follow the official documentation for poetry about tox and use a testenv like this

[testenv]
whitelist_externals = poetry
commands =
    poetry install -v
    poetry run pytest tests/
deps = poetry

I added deps = poetry as I do not have poetry globally installed, as I do not use it (except when trying to help users using tox :-) ).

e) You will not use poetry to run tox to run pytest, but you will run tox to run poetry to run pytest.

So with the above testenv, you could run the Python 3.9 tests with:

tox -e py39

Python 3.10

tox -e py310

linters

tox -e lint

P.S.: as a bonus, you could consider to use pre-commit to run all your linters: https://pre-commit.ci/

P.P.S.: I am one of the tox maintainer, I never used poetry before, but I hope this helps to get you going.