Weird behavior when using Singleton metaclass in Python unittest TestCase mixin

Question

Background

We have a special setup for which I want to write a TestCase base class / mixin that can be used to provide functionality out-of-the-box for every TestCase. The class which provides functionality to the TestCase needs to be a single instance and is therefore implemented as singleton.

---

Observation

When making it a singleton,

• the test discovery no longer properly works and only detects a single test of the TestCase
• and the single discovered test is executed N times, where N is the number of tests in the TestCase.

This is a very strange behavior I currently can not explain.

---

Reproduction

I did test a minimal working example (MWE) to reproduce this issue and it is reproducible with the following minimal code. Note that I tried to minimize as much as possible. The code therefore does not have any other purpose than this reproduction and does not follow any coding standards.

test_reproduction.py

import unittest
from unittest import TestCase


class Singleton(type):

    _instances = {}

    def __call__(cls, *args, **kwargs):
        if cls not in cls._instances:
            cls._instances[cls] = super(Singleton, cls).__call__(*args, **kwargs)
        return cls._instances[cls]


class SpecialMethods(metaclass=Singleton):

    pass


class SpecialTestCase(TestCase, SpecialMethods):

    def __init__(self, methodName="runTest") -> None:
        TestCase.__init__(self, methodName)
        SpecialMethods.__init__(self)


class TestSomething(SpecialTestCase):

    def test_something1(self):
        print('print test_something1')
        self.assertTrue(True)

    def test_something2(self):
        print('print test_something2')
        self.assertTrue(True)

    def test_something3(self):
        print('print test_something3')
        self.assertTrue(True)


if __name__ == "__main__":
    unittest.main()

When executed with: python3 -m unittest -v test_reproduction this yields the following output:

test_something1 (test_reproduction.TestSomething) ... print test_something1
ok
test_something1 (test_reproduction.TestSomething) ... print test_something1
ok
test_something1 (test_reproduction.TestSomething) ... print test_something1
ok

----------------------------------------------------------------------
Ran 3 tests in 0.000s

OK

Notice that only test_something1 is executed but three times. test_something2 and test_something3 are not executed.

---

Question

Does anyone have an explanation for what is going on? And any chance that I can make it work via Mixins (I already have other solutions, I'm only interested in making it work via Mixin).

Answer 1

There are a few things that feel wrong with your design - but you are getting what you are asking for: you create an ICBM, complete with multi-ogives to have a "singleton" by using a metaclass to create a singleton. Than, that is what you get.

We usually don't need to care how many instances or what does unittest does with the test classes we create for running all the tests: but it is obvious that you modified simple and common class behavior, that just make it break. In other words: unittest obviously have internal mechanisms to create a new instance of a class for each test it will run, and this does not work with your arrangement.

Instead of trying to figure out the inner workings of unittest, and how it marks in each instance which test it is running, it is better just to move away from your mess.

First: singletons are overrated. Metaclasses for creating a singleton are doubly overrated and wrong.

Second: multiple inheritance is good in very specific cases. Wanting to combine a class that "must be a singleton" with a class that needs multiple instances to run the tests, obviously, is not one of those cases. Use composition instead.

So, start with:

class SpecialMethods:  # <- that, just create the class

    def my_special_method_1(self):
        # do special things
    ...

SpecialMethods = SpecialMethods()  # <- here is your singleton. 
                                   # no one is creating another instance of this by accident

class TestSomething(TestCase):  # no even need for a intermediate class or a custom __init__

    def test_something1(self):
        print('print test_something1')
        SpecialMethods.my_special_method1()  # <surprise: this just works!!
        self.assertTrue(True)

As a final, not fully related tip: maybe you could use "pytest" for your tests instead of unittest: tests needs a lot less boiler-test code, including there is no need for the creation of artificial classes just to name-space the test functions.