Best practices using python mock for testing functions within sub modules

Question

So, consider I have a simple library that I am trying to write unit-tests for. This library talks to a database and then uses that data to call an SOAP API. I have three modules, and a testfile for each module.

dir structure:

./mypkg
    ../__init__.py
    ../main.py
    ../db.py
    ../api.py

./tests
    ../test_main
    ../test_db
    ../test_api

Code:

#db.py
import mysqlclient
class Db(object):
    def __init__(self):
        self._client = mysqlclient.Client()

    @property
    def data(self):
        return self._client.some_query()


#api.py
import soapclient
class Api(object):
    def __init__(self):
        self._client = soapclient.Client()

    @property
    def call(self):
        return self._client.some_external_call()


#main.py
from db import Db
from api import Api

class MyLib(object):
    def __init__(self):
        self.db = Db()
        self.api = Api()

    def caller(self):
        return self.api.call(self.db.data)

Unit-Tests:

#test_db.py
import mock
from mypkg.db import Db

@mock.patch('mypkg.db.mysqlclient')
def test_db(mysqlclient_mock):
    mysqlclient_mock.Client.return_value.some_query = {'data':'data'}
    db = Db()
    assert db.data == {'data':'data'}


#test_api.py
import mock
from mypkg.api import Api

@mock.patch('mypkg.db.soapclient')
def test_db(soap_mock):
    soap_mock.Client.return_value.some_external_call = 'foo'
    api = Api()
    assert api.call == 'foo'

In the above example, mypkg.main.MyLib calls mypkg.db.Db() (uses third-party mysqlclient) and then mypkg.api.Api() (uses third-party soapclient)

I am using mock.patch to patch the third-party libraries to mock my db and api calls in test_db and test_api separately.

Now my question is, is it recommended to patch these external calls again in test_main OR simply patch db.Db and api.Api? (this example is pretty simple, but in larger libraries, the code becomes cumbersome when patching the external calls again or even using test helper functions that patch internal libraries).

Option1: patch external libraries in main again

#test_main.py
import mock
from mypkg.main import MyLib

@mock.patch('mypkg.db.mysqlclient')
@mock.patch('mypkg.api.soapclient')
def test_main(soap_mock, mysqlcient_mock):
    ml = MyLib()
    soap_mock.Client.return_value.some_external_call = 'foo'
    assert ml.caller() == 'foo'

Option2: patch internal libraries

#test_main.py
import mock
from mypkg.main import MyLib

@mock.patch('mypkg.db.Db')
@mock.patch('mypkg.api.Api')
def test_main(api_mock, db_mock):
    ml = MyLib()
    api_mock.return_value = 'foo'
    assert ml.caller() == 'foo'

Answer 1

mock.patch creates a mock version of something where it's imported, not where it lives. This means the string passed to mock.patch has to be a path to an imported module in the module under test. Here's what the patch decorators should look like in test_main.py:

@mock.patch('mypkg.main.Db')
@mock.patch('mypkg.main.Api')

Also, the handles you have on your patched modules (api_mock and db_mock) refer to the classes, not instances of those classes. When you write api_mock.return_value = 'foo', you're telling api_mock to return 'foo' when it gets called, not when an instance of it has a method called on it. Here are the objects in main.py and how they relate to api_mock and db_mock in your test:

Api is a class                     : api_mock
Api() is an instance               : api_mock.return_value
Api().call is an instance method   : api_mock.return_value.call
Api().call() is a return value     : api_mock.return_value.call.return_value

Db is a class                      : db_mock
Db() is an instance                : db_mock.return_value
Db().data is an attribute          : db_mock.return_value.data

test_main.py should therefore look like this:

import mock
from mypkg.main import MyLib

@mock.patch('mypkg.main.Db')
@mock.patch('mypkg.main.Api')
def test_main(api_mock, db_mock):
    ml = MyLib()

    api_mock.return_value.call.return_value = 'foo'
    db_mock.return_value.data = 'some data' # we need this to test that the call to api_mock had the correct arguments.

    assert ml.caller() == 'foo'
    api_mock.return_value.call.assert_called_once_with('some data')

The first patch in Option 1 would work great for unit-testing db.py, because it gives the db module a mock version of mysqlclient. Similarly, @mock.patch('mypkg.api.soapclient') belongs in test_api.py.

I can't think of a way Option 2 could help you unit-test anything.

Edited: I was incorrectly referring to classes as modules. db.py and api.py are modules