Run tests with a set of marks, which selected based on command line parameter's value

Question

I have following tests:

@pytest.mark.hardware
@pytest.mark.feature1
@pytest.mark.feature2
def test_device1():
    pass

@pytest.mark.hardware
@pytest.mark.feature1
def test_device2():
    pass

@pytest.mark.hardware
@pytest.mark.feature2
def test_device3():
    pass

Aim: If I specify on command line argument: pytest --device device1 , I want that only tests with marks feature1, feature2, hardware will be run. Similarly: argument device2 will only evoke tests with marks hardware and feature1 etc. And if there is no argument, tests without marks will be run.

In the conftest.py I have:

def pytest_addoption(parser):
    group = parser.getgroup("filter")
    group.addoption(
        "--device",
        action="store",
        help="specify the device",
    )

I have found that pytest_collection_modifyitems could possibly help, but I don't know how to select list of marks to be run based on command line parameter's value. Thanks in advance for any help.

I have tried, but didn't work:

def pytest_collection_modifyitems(config, items):
    if config.getoption("--device") == 'device2':
        for item in items:
            item.add_marker(pytest.mark.hardware)
            item.add_marker(pytest.mark.feature1)

Answer 1

You have to filter the items list based on the device option condition. Example impl:

def marker_names(item):
    return set(m.name for m in item.iter_markers())


def pytest_collection_modifyitems(config, items):
    device_markers = {
        "device1": {"feature1", "feature2", "hardware"},
        "device2": {"feature1", "hardware"},
    }

    device_option = config.getoption("--device")

    if device_option is None:
        items[:] = [item for item in items if not list(item.iter_markers())]
    else:
        allowed_markers = device_markers[device_option]
        items[:] = [item for item in items if marker_names(item) == allowed_markers]

You can also skip tests instead of leaving them out. Example impl:

def marker_names(item):
    return set(m.name for m in item.iter_markers())


def pytest_collection_modifyitems(config, items):
    device_markers = {
        "device1": {"feature1", "feature2", "hardware"},
        "device2": {"feature1", "hardware"},
    }

    device_option = config.getoption("--device")

    if device_option is None:
        for item in items:
            if list(item.iter_markers()):
                item.add_marker(pytest.mark.skip("has no markers"))
    else:
        allowed_markers = device_markers[device_option]
        for item in items:
            if marker_names(item) != allowed_markers:
                item.add_marker(pytest.mark.skip(f"has some markers missing for {device_option}"))