getting no develop install with `pip install -e .` unless I delete `pyproject.toml`

Question

I have the following situation that pip install -e . does not build the develop version unless I delete the pyproject.toml which does not contain packaging information, but black configuration.

Does somebody know what to do in order to get the develop build.

my pyproject.toml looks like this:

# Source https://github.com/psf/black#configuration-format
[tool.black]
line-length = 100
target-version = ['py38']
exclude = '''

'''

setup.py

from setuptools import find_namespace_packages
from setuptools import setup

setup(
    name="webservice",
    packages=find_packages(),
    version="0.1.0",
    description="description",
    author="Author",
    license="License",
)

running pip install -e . with these two files...

(webservice_tool)pk@LAP1:~/webservice_tool$ pip install -e .
Obtaining file:///home/pk/webservice_tool
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: webservice
  Building editable for webservice (pyproject.toml) ... done
  Created wheel for webservice: filename=webservice-0.1.0-0.editable-py3-none-any.whl size=4070 sha256=dcb7c034ba437503d1059fe9370ccafbda144cd19f3e5d92340a63a7da396422
  Stored in directory: /tmp/pip-ephem-wheel-cache-6iqiqbob/wheels/e6/b5/ba/40d8c3d66df94ee2ae46e181034e0c3c47132784db53284d0b
Successfully built webservice
Installing collected packages: webservice
Successfully installed webservice-0.1.0

I delete pyproject.toml and only then Running setup.py develop shows up.

(webservice_tool) pk@LAP1:~/webservice_tool$ pip install -e .
Obtaining file:///home/pk/webservice_tool
  Preparing metadata (setup.py) ... done
Installing collected packages: webservice
  Attempting uninstall: webservice
    Found existing installation: webservice 0.1.0
    Uninstalling webservice-0.1.0:
      Successfully uninstalled webservice-0.1.0
  Running setup.py develop for webservice
Successfully installed webservice-0.1.0

versions of some selected packages from my conda env, running within wsl2

packaging                 21.3               pyhd3eb1b0_0  
pip                       22.1.2           py38h06a4308_0  
python                    3.8.13               h12debd9_0  
setuptools                61.2.0           py38h06a4308_0  

folder structure

|-- data_utils
|   |-- clean_cache.py
|   `-- advanced_utils.py
|-- deployment
|   |-- base_deployment
|   |   |-- auth-proxy.yaml
|   |   |-- kustomization.yaml
|   |   |-- webapi.yaml
|   |   `-- webui.yaml
|   `-- mysql_from_helm
|       |-- mysql-from-helm.yaml
|       `-- mysql-kustomization.yaml
|-- docker-compose.yml
|-- Dockerfile
|-- environment.yml
|-- live_api
|   |-- definitions.json
|   |-- __init__.py
|   `-- live_api.py
|-- params.py
|-- pyproject.toml
|-- setup.py
|-- shared_helpers
|   |-- data_cleaning.py
|   |-- handle_time.py
|   |-- __init__.py
|   |-- plot_timesequence.py
|   |-- read_samples.py
|   `-- save_samples.py
|-- setup.py
|-- util.py
|-- webtool
|   |-- clean_data
|   |   |-- clean_data.py
|   |   `-- __init__.py
|   |-- evaluation
|   |   |-- draw_figures.py
|   |   |-- __init__.py
|   |   `-- webtool_metrics.py
|   |-- __init__.py
|   |-- preprocess
|   |   |-- __init__.py
|   |   `-- preprocess.py
|   |-- ui
|   |   |-- __init__.py
|   |   `-- create_ui.py
|   `-- util
|       |-- data_input.py
|       |-- data_redefinitions.py
|       `-- __init__.py
|-- webtool.egg-info
|   |-- dependency_links.txt
|   |-- entry_points.txt
|   |-- PKG-INFO
|   |-- SOURCES.txt
|   `-- top_level.txt
`-- webtool_tests
    |-- clean_data
    |   `-- test_clean_data.py
    |-- evaluation
    |   `-- test_draw_figures.py
    |-- preprocess
    |   `-- test_preprocess.py
    `-- util
        |-- test_data_input.py
        `-- test_data_redefinitions.py

Answer 1

These are both development installs. The difference in the pip output here is because the presence (or absence) of a pyproject.toml file causes pip to use the build backend hooks (or not). From PEP 517:

If the pyproject.toml file is absent ... the source tree is not using this specification, and tools should revert to the legacy behaviour of running setup.py

You can also control that with a pip command line option:

$ pip install --help | grep pep
  --use-pep517                Use PEP 517 for building source distributions
                              (use --no-use-pep517 to force legacy behaviour).

With a PEP 517 style build, pip is setting up a dedicated environment and freshly installing setuptools for the purposes of build and/or packaging behind the scenes - see "Installing build dependencies ... done" in the log. Without it, python setup.py develop is invoked directly where it is just assumed that an adequate setuptools version is already installed within the Python runtime which was used to execute the setup.py file. The point here is that using a PEP 517 style build system allows the project to specify the setuptools version it requires (or, indeed, to use some other build system entirely).

The end result will be the same - a .pth path configuration file placed in site-packages will expose the source directory as a development installation.

Since util.py is not contained in any package, you'll also need to list this module alongside find_packages() in the setup call for it to be picked up in the development installation (as opposed to just importing from the current working directory):

# in setup.py 
from setuptools import setup

setup(
    name="webservice",
    version="0.1.0",
    packages=find_packages(),
    py_modules=["util"],  # <--- here
    ...
)