Search code examples
python-3.xsetuptoolspython-packagingrequirements.txtpyproject.toml

Setuptools unable to read requirements.txt from pyproject.toml in python 3.12.x

I have a package structure as follows:

program_root/
    - src/
    - tests/ 
    - data/
        - templates/
    - docs/         
    - pyproject.toml        
    - README
    - requirements.txt

My pyproject.toml:

[build-system]
requires = ["setuptools==62.6.0", "wheel"]
build-backend = "setuptools.build_meta"

[tool.setuptools]
package-dir = {"" = "src"}

[tool.setuptools.packages.find]
where = ["src"]

[tool.setuptools.package-data]
"*" = ["*.*"]

[project]
name = "file_ops"
version = "1.0.0"
authors = [
  {name="abc", email="[email protected]" },
]
dynamic = ["dependencies"]

[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}

when I am trying to build the package, I keep getting the error.

I am getting the following error:
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
The reason why I specifically used the setuptools version 62.6.0 is that it's the version in which setuptools introduced file attribute as following:

dependencies = {file = ["requirements.txt"]}

Now , when I have gone through the release notes of py 3.12, I got to know that ImpImporter is deprecated.

So, Using some other versions of setuptools gives one of the following errors:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte 

Backend subproccess exited when trying to invoke get_requires_for_build_sdist

I am clueless on what version of setuptools should I use now to build the package successfully.

This setuptools history (2nd Point) says that few fields are still in beta. how to know what are they exactly?

Please help me understand if I am doing anything wrong here.

BTW, I am strongly inclined to use only toml file for my packaging needs.

All of the traceback for my first error:

* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - setuptools==62.6.0
  - wheel
* Getting build dependencies for sdist...
Traceback (most recent call last):
  File "D:\VAMSI-vectra\Vectra_work\jsonMerging\shared_libraries\vct_file_ops_root\venv_file_ops\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 373, in <module>
    main()
  File "D:\VAMSI-vectra\Vectra_work\jsonMerging\shared_libraries\vct_file_ops_root\venv_file_ops\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 357, in main        
    json_out["return_val"] = hook(**hook_input["kwargs"])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\VAMSI-vectra\Vectra_work\jsonMerging\shared_libraries\vct_file_ops_root\venv_file_ops\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 302, in get_requires
_for_build_sdist
    backend = _build_backend()
              ^^^^^^^^^^^^^^^^
  File "D:\VAMSI-vectra\Vectra_work\jsonMerging\shared_libraries\vct_file_ops_root\venv_file_ops\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 70, in _build_backen
d
    obj = import_module(mod_path)
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Vectra\AppData\Local\Programs\Python\Python312\Lib\importlib\__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "C:\Users\Vectra\AppData\Local\Temp\build-env-kr3vu6_o\Lib\site-packages\setuptools\__init__.py", line 16, in <module>

    import setuptools.version
  File "C:\Users\Vectra\AppData\Local\Temp\build-env-kr3vu6_o\Lib\site-packages\setuptools\version.py", line 1, in <module>  
    import pkg_resources
  File "C:\Users\Vectra\AppData\Local\Temp\build-env-kr3vu6_o\Lib\site-packages\pkg_resources\__init__.py", line 2191, in <mo
dule>
    register_finder(pkgutil.ImpImporter, find_on_path)
                    ^^^^^^^^^^^^^^^^^^^
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_sdist
(venv_file_ops) PS D:\VAMSI-vectra\Vectra_work\jsonMerging\shared_libraries\vct_file_ops_root

Few resources that helped me while I am trying to resolve:

  1. installing setup tools, wheel in venv. But, https://docs.python.org/3/whatsnew/3.12.html#ensurepip says that in pip > 22.x no need of such explicit installation

  2. Refer to this SOQ: How to reference a requirements.txt in the pyproject.toml of a setuptools project? pointing to requirements.txt is introduced in version 62.6.0 Some features are still in beta version https://setuptools.pypa.io/en/latest/history.html#v68-1-0

  3. Version of setuptools that introduced mentioning 'file' in dependencies https://github.com/pypa/setuptools/blob/v62.6.0/CHANGES.rst#v6260 When I used the same version in the [build-system] table, I was getting ImpImporter is unavailable --issue is incompatibiltiy of setuptools v67.6.0 with Py 3.12 because ImpImporter is deprecated in 3.12 -- refer to this release note : https://docs.python.org/3/whatsnew/3.12.html#importlib

  4. Setuptools release history notes https://setuptools.pypa.io/en/latest/history.html

Edit: I got to understand from the inputs of @9769953 that the issue is with encoding of my windows system. Having an UTF-8 encoded requirements file solved the issue. Now, the issue is configuring the pip freeze > requriements.txt command to ensure that it writes the file in UTF-8 (Not sure if that's possible) OR Modifying the Default encoding of my PC to UTF-8

Solution

  • From the inputs of @9769953, I am able to understand the issue. While generating requirements file with the command pip freeze > requirements.txt, the file is being encoded in UTF-16 LE (In your case, it might be different). so, if the encoding is not UTF-8, chances are that it might be the root cause.

    I have manually copied my requirements file into Notepad and in Encoding Tab, I have selected UTF-8 ->saved and moved the file to actual location (location where the requirements file is supposed to be) and then the issue resolved.

    So, to programmatically configure the encoding of pip freeze command, I've used the following: python pip freeze | python -c "import sys, codecs; sys.stdout = codecs.open('requirements.txt', 'w', 'utf-8'); print(sys.stdin.read())"