Running all test files in python from one function using "assert" without exiting "for loop" on test failure

Question

I am working on test runner which reads ".test" file from a given directory.

Structure of ".test" is:

---TEMPLATE---
template content
---CONTEXT---
context to render on template
---RESULT---
expected result of the template.

There are 'n' no. of test file in my test directory. I stores test no. of .test files in dictionary "tests" as its key and name of .test file as its value.

after that iterate on dictionary "tests" and read the content of .test file and store them in variables.

---TEMPLATE--- part in "template_string",
---CONTEXT--- part in "context_string", and
---RESULT--- part in "expected_result"

then render template_string with context_string using jinja2.Environment class and store them in "result" varibale.

compare "result" with "expected_result".

current test runner code:

class TestTempates(TestCase):
     def test_method(self):
         tests = { dictionary of .test file }
         results = {} #to store status of test case at there index (pass or error).
         env = jinja2.Environment()
         passed = 0
         error = 0    
         for index, fname in tests.items():
             file_path = dirpath + os.sep + fname
             with open(file_path) as f:
                 # logic to read file content 
                 template_string = #content of ---TEMPLATE--- part from file
                 context_string = #content of ---CONTEXT--- part from file
                 expected_result = #content of ---RESULT--- part from file
             template = env.from_string(template_string)
             context = json.loads(context_string)
             result = template.render(context)

             if result == expected_result:
                 results[index] = "Pass"
                 passed += 1
             else:
                 sep = "-----------------------------"
                 error = "Error: results mismatch:\n%s\n%s\n%s\n%s" % \
                         (sep, result, sep, expected_result)
                 results[index] = error
                 errors += 1

comparing "result" and "expected_result" with in "if else" condition is working fine. But now I want to use "assert" or "assertEquals" without exiting from "for loop" when any test file "result" doesn't matched with "expected_result" until all tests files are not executed. So, that I can use my test runner in Travis CI so that Travis build will failed when any test case get failed.

In current situation Travis CI build is not failing on test case failure.

Answer 1

You can follow below code of snippet to solve your problem.

suite = unittest.TestSuite()

def test_main(self):
    self.assertEquals(self.result, self.expected_output)


def test_method(self):
    """
    """
    # -- code to get tests objects to have all .tests content
    for index, fname in tests.items():
        # get result and expected_output value
        obj = type('Test', (unittest.TestCase,), {'test_main': test_main,
                  'result':result, 'expected_output':expected_output})

        suite.addTest(obj('test_main'))

unittest.TextTestRunner(verbosity=2).run(suite)