near-useless assertion output from hamcrest.contains_inanyorder applied to unittest.mock.Mock.mock_calls

Question

Often, I care about the exact calls the system under test makes to another part of the software (which I mock in the test), but not about the order, in which those calls happen. (E.g. because the end effect on the real other part replaced by the mock does not depend on the order of these calls.)

In other words, I want my test to

• fail if not all expected calls have been made
• fail if unexpected calls have been made (so unittest.mock.Mock.assert_has_calls does not suffice)
• not fail if only the order of the calls changed
• fail if a call has been made less or more often than expected

So, I have to inspect the mock_calls property of the mock object. I can do that in a generic and reasonably comprehensible way with PyHamcrest's contains_inanyorder:

#!/usr/bin/env python3
from unittest import TestCase, main
from unittest.mock import Mock, call
from hamcrest import assert_that, contains_inanyorder as contains_in_any_order

class TestMockCalls(TestCase):
    def test_multiple_calls(self):
        m = Mock()
        m('foo')
        m.bar('baz')
        m('foo')
        assert_that(
            m.mock_calls, contains_in_any_order(
                call('foo'),
                call('foo'),
                call.bar('baz'),
            )
        )

if __name__ == '__main__':
    main()

This works fine for passing tests, like the one above:

$> ./test_mock_calls.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

It also fails when it should fail (as specified above, e.g. when you change one of the m('foo') to m('F00')), but the output in that case is not as useful as it could be:

$> ./test_mock_calls.py
F
======================================================================
FAIL: test_multiple_calls (__main__.TestMockCalls)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_mock_calls.py", line 16, in test_multiple_calls
    call.bar('bay'),
AssertionError: 
Expected: a sequence over [, , ] in any order
     but: not matched: 


----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)

The only information (apart from which test and which assertion failed) I can gather from this, is how many calls on the mock were expected in total (by counting the commas between the square brackets), but not what calls were expected and, more importantly, what and how many calls were actually observed.

Is this a bug in unittest.mock or PyHamcrest or am I using them wrong?

Answer 1

The problem is that call (_Call) itself if a kind of mock, and overrides __getattr__. When hamcrest starts checking whether it has a decribe_to attribute, things start going wrong.

I think that since both modules are doing introspective things, no single one is to blame, and special cases should be implemented on either side to play well with the other (probably in hamcrest, since mock is a standard module).

A user-side workaround is to do:

from unittest.mock import _Call
_Call.describe_to = lambda c, d: d.append(str(c))