Search code examples
pythoncontinuous-integrationtox

Running multiple tox testenv matching a name fragment

Rationale

With complex dependency matrix, tox testenv name patterns end up being a list like

py37-pytest5-framework1
py37-pytest5-framework2
py37-pytest6-framework1
py37-pytest6-framework2
py38-pytest5-framework1
py38-pytest5-framework2
py38-pytest6-framework1
py38-pytest6-framework2
...
py310-pytest6-framework2

While the inner tox.ini syntax allows configuring a lot of things with name fragments, e.g.

[testenv]
basepython =
    py37: python3.7
    py38: python3.8
    py39: python3.9
    py310: python3.10
deps =
    pytest5: pytest ~= 5.0
    pytest6: pytest ~= 6.0
    framework1: framework ~= 1.0
    framework2: framework ~= 2.0
setenv =
    framework2: FOO=bar

I find there is no way in telling the tox CLI in running all testenvs matching a name fragment like tox -e py39 or tox -e framework2.

Issues

The main drawback is that most usually CI testing jobs will end up being segregated by python version, so you end up writing instructions like

tox -e $PY-pytest5-framework1,$PY-pytest5-framework2,$PY-pytest6-framework1,$PY-pytest6-framework2

but then the CI jobs definition is coupled to the tox test matrix because it must be aware of:

  • testenvs being added or removed
  • matrix exclusions like pytest-5 is not compatible with python-3.10

And this is cumbersome to maintain.

Incomplete workaround

An easy-to-go workaround is simply running tox --skip-missing-interpreters, but the drawbacks are:

  • CI jobs can't be segregated by framework version instead of python version, for example to reuse some special framework cache
  • CI VMs could feature system python installations beyond the one targeted by each job, so you could en up with e.g. python-3.8 being run in all CI jobs.

Question

Am I missing some out-of-the-box mechanism to filter the testenvs to be run with a fragment that powers me to write CI jobs agnostic to the tox dependency matrix? I mean something like tox -e '*-framework2'.

Am I bound to filter and aggregate the output of tox --listenvs with shell tricks?

Solution

  • You could negate a regex pattern for the TOX_SKIP_ENV as the following:

    $ env TOX_SKIP_ENV='.*[^-framework2]$' tox

    tox4, which will be introduced within the next couple of months, introduces labels. While this may be not an immediate help for your problem, maybe you see a way to simplify your tox.ini.