Using python venv with custom libraries

Question

I'm working in a Linux environment and using a venv. I come from C++ and structured the whole project assuming the existence of something akin to CMake. Basically my idea was to keep well separated the "libraries" from their respective tests, and when the libraries are fully developed to call them inside of a main program. I built it like this:

mainFolder:

- .venv

- devel (containing the tests)

-- devel_modules1

-- devel_modules2

-- devel_modules3

- modules (containing the modules)

-- modules1

-- modules2

-- modules3

Each of devel_modules1-3 contains several python files to test the corresponding file in modules1-3; I would like to be able to use an import statement "from anywhere" as long as I'm using the venv. My understanding is that to do this I should treat modules as a package, install it with pip and it should just work (???).

In each of modules1-3, I placed a file __init__.py containing

`__all__=[list of all the python files in this folder]`

In modules, I placed an empty __init__.py file and a file setup.py containing :

from setuptools import setup, find_packages
import pathlib
here = pathlib.Path(__file__).parent.resolve()

setup(
    name="modules",
    version="0.0.1",
    packages=find_packages(where="src")
) 

When I launch

pip3 install -e modules 

It works without raising any error and is correctly listed in the installed packages;

But when I try to launch someTest.py contained in devel_modules I get the error

ModuleNotFoundError: No module named 'modules' 

from the line

from modules.modules1.fileModules1 import DatabaseConnector 

What am I doing wrong?

Answer 1

This is the project structure I would recommend:

modules
├── .venv
│   └── ...
├── pyproject.toml
├── src
│   └── modules
│       ├── __init__.py
│       ├── modules1
│       │   ├── __init__.py
│       │   ├── some_module.py
│       │   └── ...
│       └── modules2
│           ├── __init__.py
│           └── another_module.py
└── test
    ├── test_modules.py
    ├── modules1
    │   └── test_modules1.py
    └── modules2
        └── test_modules2.py

pyproject.toml

[build-system]
build-backend = 'setuptools.build_meta'
requires = [
    'setuptools',
]

[project]
name = "modules"
version = "0.0.1"

This is a "canonical" project structure. There should be no need for setup.py or setup.cfg. The src-layout is automatically recognized by setuptools, this layout protects from some common import issues during development phase when used in combination with the "editable" installation (.venv/bin/python -m pip install --editable .). The tests can be nested as much as needed, a "test runner" like pytest should be able to handle this without issues.