Python unittest: Run multiple assertions in a loop without failing at first one, but continue

Question

Scenario: one of my test cases is executing a shell program with a couple of input files and a specific output. I'd like to test different variations of these input/output and each of these variations is saved in its own folder, i.e. folder structure

/testA
/testA/inputX
/testA/inputY
/testA/expected.out
/testB
/testB/inputX
/testB/inputY
/testB/expected.out
/testC/
...

Here's my code for running this test:

def test_folders(self):
    tfolders = glob.iglob(os.environ['testdir'] + '/test_*')                
    for testpath in tfolders:
        testname = os.path.basename(testpath)
        self.assertTrue(self.runTest(testname))

testname in this case is "testX". It executes the external program with inputs in that folder and compares it to expected.out in that same folder.

Problem: As soon as it hits a test that fails, the testing stops and get:

Could not find or read expected output file: C:/PROJECTS/active/CMDR-Test/data/test_freeTransactional/expected.out
======================================================================
FAIL: test_folders (cmdr-test.TestValidTypes)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\PROJECTS\active\CMDR-Test\cmdr-test.py", line 52, in test_folders
    self.assertTrue(self.runTest(testname))
AssertionError: False is not true

Question: How do I make it continue the rest of the tests and show how many failed? (in essence dynamically create a unittest and execute it)

Thanks

Answer 1

How about something like this?

def test_folders(self):
    tfolders = glob.iglob(os.environ['testdir'] + '/test_*')    

    failures = []    
    for testpath in tfolders:
        testname = os.path.basename(testpath)
        if not self.runTest(testname):
            failures.append[testname]

    self.assertEqual([], failures)