Using pytest with "dynamic" test names

Question

I'm trying to create multiple tests, using two dictionaries. Below, expected means my expected outputs, and actual means my actual outputs.

import pytest
from dataclasses import dataclass
from typing import Dict


@dataclass
class OutputsToCheck:
    a: int
    b: int


expected: Dict[str, OutputsToCheck] = {
    "test_1": OutputsToCheck(a=1, b=2),
    "test_2": OutputsToCheck(a=3, b=4),
}

actual: Dict[str, OutputsToCheck] = {
    "test_1": OutputsToCheck(a=1, b=2),
    "test_2": OutputsToCheck(a=3, b=4),
}

I want pytest to report success/failures for each item in the respective dictionaries, but I'm not able to achieve this.

One thing I've tried is this:

@pytest.fixture
def test_names():
    return ["test_1", "test_2"]


@pytest.fixture
def expected_outputs():
    return expected


@pytest.fixture
def actual_outputs():
    return actual


@pytest.fixture
def make_test(expected_outputs, actual_outputs):
    def test(test_name: str):
        expected = expected_outputs[test_name]
        actual = actual_outputs[test_name]
        assert expected == actual

    return test


@pytest.fixture
def make_tests(make_test, test_names):
    output = {}
    for name in test_names:
        output[name] = make_test(name)
    return output


def test(make_tests, test_names):
    for name in test_names:
        make_tests[name]

But this just tells me that "1 test has passed". I'd like pytest to say that "2 tests have passed".

Is there a way of achieving this? Many thanks.

Answer 1

In short, the canonical way to do something like this in Pytest is the parametrize mark.

I'll illustrate the most verbose form, which lets you also set a custom id for the parameter set (to customize (part of) the test name):

import pytest


@pytest.mark.parametrize(("a", "b", "expected_output"), [
    pytest.param(1, 2, 3, id="three"),
    pytest.param(4, 9, 13, id="thirteen"),
])
def test_something(a, b, expected_output):
    assert a + b == expected_output

When you run this, you'll get something like

collecting ... collected 2 items

so78513157.py::test_something[three] PASSED
so78513157.py::test_something[thirteen] PASSED

2 passed in 0.02s

Note how the id is shown in the brackets after the test.

You can also add multiple parametrize marks, to get a cartesian product of all of the parameter combinations. (This is showing a shorter form of parametrize, too.)

import pytest


@pytest.mark.parametrize(("a", "b", "expected_output"), [
    pytest.param(1, 2, 3, id="three"),
    pytest.param(4, 9, 13, id="thirteen"),
])
@pytest.mark.parametrize("way", ["this way", "that way"])
def test_something(a, b, expected_output, way):
    if way == "this way":
        assert a + b == expected_output
    else:
        assert b + a == expected_output

This will show

so78513157.py::test_something[this way-three] PASSED
so78513157.py::test_something[this way-thirteen] PASSED
so78513157.py::test_something[that way-three] PASSED
so78513157.py::test_something[that way-thirteen] PASSED

so we got 2 x 2 = 4 tests.

You can extend this idea to use dataclasses or namedtuples for your parameters if that makes sense for your tests.