Django tests - print all test failures for a series of n times the same test

Question

I'd like that every assertion test in a TestCase is actually tested, even if the first one fails. In my situation, all the assertions are of the same nature.

Actually I have something that evaluates formulas written as Python objects (figure it as formulas written as strings to be eval'd). I'd like to do something like:

class MyTest(TestCase):
   def test_something(self):
       for id in ids:
           for expression in get_formulas(id):
                for variable in extract_variables(expression):
                    self.assertIn(variable, list_of_all_variables)

=> I want to see printed all of the variables that are not in the list_of_all_variables!

This is necessary for me to review all my so-called formulas and be able to correct errors.

Some more context:

I'm having a variable number of tests to perform (depending on a list of IDs written in a versioned data file) in one app.

To have a variable number of TestCase instances, I did write a base class (mixin), then build on-the-fly classes with the use of 3-args type function (that is creating classes).

This way, I have n tests, corresponding to the n different ids. It's a first step, but what I want is that each and every assertion in those tests gets tested and the corresponding assertion errors get printed.

Answer 1

As referenced in the question Continuing in Python's unittest when an assertion fails, failing at assertion errors is the hardcoded behavior of the TestCase class.

So instead of changing it's behavior, I generated a lot of different test_... methods to my classes, in the following style:

from django.test import TestCase
from sys import modules


# The list of all objects against which the tests have to be performed
formids = [12,124,234]
# get_formulas returns a list of formulas I have to test independently, linked to a formid
formulas = {id: get_formulas(id) for id in formids}

current_module = sys.modules(__name__)

def test_formula_method(self, formula):
    # Does some assertions
    self.assertNotEqual(formula.id, 0)

for formid in formids:
    attrs = {'formid': formid}
    for f in formulas[formid]:
        # f=f so the 2nd arg to test_formula_method is staying local
        # and not overwritten by last one in loop
        attrs['test_formula_%s' % f.name] = lambda self, f=f: test_formula_method(self, f)

    klass_name = "TestForm%s" % formid
    klass = type(klass_name, (TestCase,), attrs)

    setattr(current_module, klass_name, klass)