write pytest test function return value to file with pytest.hookimpl

Question

I am looking for a way to access the return value of a test function in order to include that value in a test report file (similar to http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures).

Code example that I would like to use:

# modified example code from http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures

import pytest
import os.path

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    if rep.when == "call" and rep.passed:
        mode = "a" if os.path.exists("return_values") else "w"
        with open("return_values.txt", mode) as f:
            # THE FOLLOWING LINE IS THE ONE I CANNOT FIGURE OUT
            # HOW DO I ACCESS THE TEST FUNCTION RETURN VALUE?
            return_value = item.return_value
            f.write(rep.nodeid + ' returned ' + str(return_value) + "\n")

I expect the return value to be written to the file "return_values.txt". Instead, I get an AttributeError.

Background (in case you can recommend a totally different approach):

I have a Python library for data analysis on a given problem. I have a standard set of test data which I routinely run my analysis to produce various "benchmark" metrics on the quality of the analysis algorithms on. For example, one such metric is the trace of a normalized confusion matrix produced by the analysis code (which I would like to be as close to 1 as possible). Another metric is the CPU time to produce an analysis result.

I am looking for a nice way to include these benchmark results into a CI framework (currently Jenkins), such that it becomes easy to see whether a commit improves or degrades the analysis performance. Since I am already running pytest in the CI sequence, and since I would like to use various features of pytest for my benchmarks (fixtures, marks, skipping, cleanup) I thought about simply adding a post-processing hook in pytest (see http://doc.pytest.org/en/latest/example/simple.html#post-process-test-reports-failures) that collects test function run time and return values and reports them (or only those which are marked as benchmarks) into a file, which will be collected and archived as a test artifact by my CI framework.

I am open to other ways to solve this problem, but my google search conclusion is that pytest is the framework closest to already providing what I need.

Answer 1

pytest ignores test functions return value, as can be seen in the code:

@hookimpl(trylast=True)
def pytest_pyfunc_call(pyfuncitem):
    testfunction = pyfuncitem.obj
    ...
    testfunction(**testargs)
    return True

You can, however, store anything you need in the test function; I usually use the config object for that. Example: put the following snippet in your conftest.py:

import pathlib
import pytest


def pytest_configure(config):
    # create the dict to store custom data
    config._test_results = dict()


@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    # execute all other hooks to obtain the report object
    outcome = yield
    rep = outcome.get_result()

    if rep.when == "call" and rep.passed:
        # get the custom data
        return_value = item.config._test_results.get(item.nodeid, None)
        # write to file
        report = pathlib.Path('return_values.txt')
        with report.open('a') as f:
            f.write(rep.nodeid + ' returned ' + str(return_value) + "\n")

Now store the data in tests:

def test_fizz(request):
    request.config._test_results[request.node.nodeid] = 'mydata'