Proper way to deal with fixture data in Python

Question

My program is producing natural language sentences. I would like to test it properly by setting the random seed to a fix value and then:

• producing expected results;
• comparing generated sentences with expected results;
• if they differ, asking the user if the generated sentences were actually the expected results, and in this case, updating the expected results.

I already met such systems in JS, so I am surprised for not finding it in Python. How do you deal with such situations?

Answer 1

One drawback of asking the user to replace the expected answer is that the automated test can not be run automatically. Therefore, test frameworks do not allow reading from input.

I really wanted this feature, so my implementation looks like:

def compare_results(expected, results):
    if not os.path.isfile(expected):
        logging.warning("The expected file does not exist.")
    elif filecmp.cmp(expected, results):
        logging.debug("%s is accepted." % expected)
        return 
    content = Path(results).read_text()
    print("The test %s failed." % expected)
    print("Should I accept the results?")
    print(content)
    while True:
        try:
            keep = input("[y/n]")
        except OSError:
            assert False, "The test failed. Run directly this file to accept the result"
        if keep.lower() in ["y", "yes"]:
            Path(expected).write_text(content)
            break
        elif keep.lower() in ["n", "no"]:
            assert False, "The test failed and you did not accept the answer."
            break
        else:
            print("Please answer by yes or no.")


def test_example_iot_root(setup):
    ...
    compare_results(EXPECTED_DIR / "iot_root.test", tmp.name)


if __name__ == "__main__":
    from inspect import getmembers, isfunction
    def istest(o):
        return isfunction(o[1]) and  o[0].startswith("test")

    [random.seed(1) and o[1](setup) for o in getmembers(sys.modules[__name__]) \
            if istest(o)]

When I run directly this file, it asks me whether or not it should replace the expected results. When I run from pytest, input creates an OSError that allows to quit the loop. Definitely not perfect.