Search code examples
pythontestingpytestabstract-syntax-treeinspect

Parsing out data from pytest assert and converting into a string


I want to collect data about test to an external database, where I would collect all assert and parse the expected value versus the actual result.

To describe what I want in code:

test_something.py:

def test_me_1(send_to_test):
    """
    Docstring to pad even more
    """
    x,y = send_to_test(3)
    assert x == 3**2
    assert y == 3**3
    assert y == 3**2 # I want this to fail intentionally


conftest.py:

@pytest.fixture(scope='function')
def send_to_test() -> Callable:
    return lambda x: (x ** 2, x ** 3)



The data I want to collect is (Any other way is fine as long as I get this data):

test_data = [{'func': 'test_me_1', 'variable': 'x', 'expected_value': 9, 'actual_value': 9},
             {'func': 'test_me_1', 'variable': 'y', 'expected_value': 27, 'actual_value': 27},
             {'func': 'test_me_1', 'variable': 'y', 'expected_value': 27, 'actual_value': 9}]

My current idea is a wrapper, that can easily be implemented to existing test and will collect the assert statements and their value.
Collecting the number of assert statements is straightforward if I use AST, but these are good only for static analysis. I've also tried to use the inspect module, but have hard time figuring out how to use it for this purpose

My poor attempt thus far (print statements are temporary since I want to collect the data)

def get_this(func: Callable):
    @functools.wraps(func)
    def wrapper(*arg, **kwargs):
        print(func.__name__)
        func_content = inspect.getsource(func)
        func_root = ast.parse(func_content)
        for node in func_root.body:
            for content in node.body:
                if isinstance(content, ast.Assert):
                    print(content)
        return func(*arg, **kwargs)

    return wrapper

Solution

  • @hoefling answered my question with quite an easy solution of using the following hooks in conftest:

    def pytest_assertrepr_compare(op, left, right):...
    def pytest_assertion_pass(item, lineno, orig, expl):...
    

    According to the Pytest documentation, I have to add to pytest.ini enable_assertion_pass_hook=true and erase .pyc files, but other than that it works like a charm. Now I have left and right comparisons with the operator used (pytest_assertrepr_compare) and if the test passed (pytest_assertion_pass).