How to configure VS Code pytest when Python code is not in top level directory?

I started a project with what I would describe as a "standard" Python project structure, whereby my code is organized into a set of packages which all live in the top level directory. In addition to this, there is a tests directory which contains .py files with functions which define the test cases.

I would now like to segregate this from another new project which I will add to the same repository.

To do this, I propose to create two top level directories new_project and old_project under which everything will be organized. (Given the names, you could imagine this is part of some kind of migration.)

I don't know how to make this work with pytest as it integrates with VS Code.

Here's the proposed project structure, as a MWE:

new_project/
    simple_package/
        __init__.py
    tests/
        test_simple_package.py

old_project/
    # copy & paste of the above

The files can contain minimal code to demonstrate the point.

$ cat new_project/simple_package/__init__.py
a = 1

$ cat  new_project/tests/test_simple_package.py
from simple_package import a

def test_a():
    assert a == 1

I see the following error message:

In the pytest Test Explorer sidebar (left hand side, beaker icon): pytest Discovery Error [pytest-subdirectory-test] Show output to view error logs
In the output: E ModuleNotFoundError: No module named 'simple_package'

Is it possible to configure pytest to work with such a repository structure?

It would seem strange that this wouldn't work, because many projects would likely be a mix of different code written in different programming languages. Therefore it would seem strange if the VS Code pytest integration can only be made to work in the most simple case where all the Python packages are at the top level of the repository.

Solution

There is another solution, which again, is a bit of a hack, and again, I don't really like it much. It involves making everything into a single package.

Here's how we do that:

Add __init__.py to new_project and old_project. This converts both directorys into Python Packages. Because these packages exist at the top level, we can import from them without having to modify PYTHONPATH.

The code in the tests needs to change slightly. The import statement becomes:

from new_project.simple_package import a

Unlike my previous solution (hack), this has the advantage that VS Code/Python plugin can automatically update the imports if we move files and folders around, or rename things. (Under some, but not all conditions.)

However I still don't like this. In particular, there are some potential problems:

I don't know how this would affect deployment, or building, of packages (if someone knows please let me know). I would guess at the very least, we will end up with one additional level of namespace
The additional level of namespace is pointless, and potentially misleading or confusing

In addition, I don't this this really solves the problem. I think we need to move the test folders to the top level as well (not 100% sure on this)

If so we end up with this structure:

new_project/
    __init__.py
    ... other
old_project/
    __init__.py
    ... other
tests/
    test_new_project/
        ... tests
    test_old_project/
        ... tests

This maybe isn't the end of the world, since the tests are still separated (from the source). But the fact that they are bundled together in the same subdirectory is potentially an issue because we can't split the project into two independent parts, whereas if the two top level directories were new_project and old_project these would truly be two independent source trees which we could split into two independent repositories.

You may ask: Since I propose to make both new_project and old_project into Packages, why bother with these at all. Just keep all the previously existing packages at the top level.

The issue with this is for a complex project with many packages, there probably won't be an obvious way to tell which packages belong to which project.

Consider this example:

mem_cache/
mem_cache_webserver/
mem_cache_cli/
data/
test_mem_cache/
expected_output/
output/
input/
test/
old_memcache/
util_io/
memcache_simulation_driver/
memcache_simulation/

Which of these are part of the new project and which belong to the old project? There is no way to tell.