Combining `runpy` with mocks passes when running alone but fails when running with other tests

Question

When I run the following test alone, it passes. But when I run it with other tests, even if this test is the first to run it fails:

from runpy import run_path
from unittest.mock import patch, MagicMock

@patch('boto3.client', autospec=True)
def test_scripts(mock_boto3_client: MagicMock):
    mock_boto3_client().head_object.return_value = {}
    run_path(
      'src/scripts/dynamic_script.py',
      run_name='__main__'
    )

This seems to somehow be affected by other mocks or imports, but it's unclear how they would affect this test when it runs first

Answer 1

Eventually figured it out. This is due to the way mock.patch works. Here's a good explanation:

https://nedbatchelder.com/blog/201908/why_your_mock_doesnt_work.html

The reason runpy made this harder to understand was that when we run only this test, the patch worked properly even though the patch wasn't on the correct name. This is probably because the patch was done before the import (the import happened dynamically during run_path), so by the time from/import happened, the patch was already there in memory and we got the pointer to the mock.

When we run the test in the context of other tests, the import tree includes the original library defined with from boto3 import client, so this import is probably in memory before the patch is applied.

The solution is either to make sure all references to the mocked entity are full-named by only using import boto3, or to mock the name inside the tested module.