Search code examples
pythonmetaprogrammingpython-unittest

Detecting testcases created dynamically while running unittest module


I've faced a problem. In my situation I create testcases dynamically like this

class TestMetrics(unittest.TestCase):
    @staticmethod
    def generate_test(test: ElementaryTestCase):
        def test_instance(self):
        ...
        self.assertLessEqual(...)
        ...
        return test_instance

and then in module's body add another generate_test function to the TestMetrics class

    tests_query = prepare_test_query()

    for test in tests_query:
        formatted_test_name = '_'.join(test.test_name.strip().split())
        test_method = TestMetrics.generate_test(test)
        test_name = f'test_{formatted_test_name}'

        setattr(TestMetrics, test_name, test_method)

It works fine when I start my tests with unittest.main(), but now my goal is to start them as it should be using python3 -m unittest test_modulename, but it obviously sees 0 tests. I want to try to itegrate it with allure then, so that's why I need this opportunity to start tests by default. Is there any solution how to make unittest see generated tests?

Example of what's going on

import unittest

class TestExample(unittest.TestCase):
    @staticmethod
    def generate_test(*args):
        def test_instance(self):
            a,b = args
            self.assertEqual(a, b)
        return test_instance

values = [
    (1, 1, 'example'),
    (12, 12, 'another_example'),
    (0, 5, 'fail_test')
]

for val in values:
    test_name = 'test_' + val[-1]
    test_method = TestExample.generate_test(*val[:2])
    setattr(TestExample, test_name, test_method)

unittest.main()

Solution

  • class TestExample(unittest.TestCase):
        @staticmethod
        def generate_test(*args):
            def test_instance(self):
    

    During discovery generate_test() never runs, so the test of interest is not found.

    But you know how to convince discovery to find and run some code. Just offer an ordinary test:

    class TestExample(unittest.TestCase):
        def test_initialize(self):
            # (call unittest.main() here, or similar
            # code which will call into generate_test() and
            # make a suitable test suite available to the runner.
    
        @staticmethod
        def generate_test(*args):
    

    what does "dynamic" mean?

    As a completely different approach, consider writing dynamically generated valid source code to tests/dynamic.py, and run discovery on that (now "static") collection of tests.