Search code examples
piplocustconfluent-kafka-pythonpython-3.11

Unable to upgrade Python 3.11 due to error in Locust-plugin installation (Confluent-Kafka)

After upgrading Python to 3.11, Locust-plugin fails to install as one of its dependency throws error

  1. First it asked me to install VC++ > 14.0
  2. After installing VC++ build tools 15, still it throws a different error

Note: We do not see this issue until Python 3.10.8

Kindly let me know if there is any fix?

locust-plugins==2.7.1
  - confluent-kafka [required: Any]

DEPRECATION: confluent-kafka is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 2
3.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for confluent-kafka ... error
  error: subprocess-exited-with-error

  × Running setup.py install for confluent-kafka did not run successfully.
  │ exit code: 1
  ╰─> [57 lines of output]
      running install
      C:\~\AppData\Local\Programs\Python\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use bu
ild and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      .............build\~\AppData\Local\Temp\pip-install-751nsega\confluent-kafka_3aae65aeec6a4e2ab93d05a4141fc55b\src
      creating build\temp.win-amd64-cpython-311\Release\~\AppData\Local\Temp\pip-install-751nsega\confluent-kafka_3aae65aeec6a4e2ab93d05a4141fc55b\src\confluent_kafka     
      creating build\~\AppData\Local\Temp\pip-install-751nsega\confluent-kafka_3aae65aeec6a4e2ab93d05a4141fc55b\src\confluent_kafka\src
      "C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD -IC:\~\AppData\
Local\Programs\Python\Python311\include -IC:\~\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\M
SVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)
\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\inIC:\~\AppData\Local\Programs\Python\Python311\include -IC:\~\AppData\Local\Programs\Python\Python311\Include "-IC:\Program Files 
(x86)\Microsoft Visual Studio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /TcC:\~\AppData\Local\Temp\pip-install-751nsega\confluent-kafka_3aae65aeec6a4e2ab93d05a4141fc55b\src\confluent_kafka\src\Admin.c /Fobuild\~\AppData\Local\Temp\pip-install-751nsega\confluent-kafka_3aae65aeec6a4e2ab93d05a4141fc55b\src\confluent_kafka\src\Admin.obj
      Admin.c      c:\~\appdata\local\temp\pip-install-751nsega\confluent-kafka_3aae65aeec6a4e2ab93d05a4141fc55b\src\confluent_kafka\src\confluent_kafka.h(23): fatal error C1083: Cannot open include file: 'librdkafka/rdkafka.h': No such file or directory
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\BuildTools\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> confluent-kafka
Solution

  • Right now there is currently no pre-built version of confluent-kafka for 3.11. Hopefully it is on the way though: https://github.com/confluentinc/confluent-kafka-python/issues/1452

    Also, I think the reason cannot build from source is that you need librdkafka installed. I dont know what the windows equivalent is, but apparently someone got it working on Linux this way:

    apt-get install -y --no-install-recommends gcc git libssl-dev g++ make && cd /tmp && git clone https://github.com/edenhill/librdkafka.git && cd librdkafka && ./configure --prefix=/usr && make && make install