Search code examples
pythonpython-3.xpipsnowflake-cloud-data-platform

Building wheel for pyarrow (pyproject.toml) did not run successfully


Just had IT install Python 3.12 on my Windows machine. I do not have admin rights on my machine, which may or may not be important. During install, the following were done:

  • Clicked "Add Python 3.n to Path" box.
  • Went into Customize installation and made sure pip was selected, and, selected "install for all users".

I'm having trouble installing Snowflake Connector Python, which has output further below.

For sanity, I tried installing numpy, which seemingly worked even though "platform independent libraries not found":

py -m pip install numpy1

Could not find platform independent libraries <prefix>
Collecting numpy1
  Using cached numpy1-0.0.1-py3-none-any.whl
Installing collected packages: numpy1
Successfully installed numpy1-0.0.1

And here's the attempt at installing the Snowflake Connector. Would anyone have ideas on my problem(s)?

#Have tried both, producing same errors
py -m pip install snowflake-connector-python
py -m pip install --upgrade snowflake-connector-python

Could not find platform independent libraries <prefix>
Collecting snowflake-connector-python
  Using cached snowflake-connector-python-3.3.0.tar.gz (716 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [326 lines of output]
      Could not find platform independent libraries <prefix>
      Collecting setuptools>=40.6.0
        Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
      Collecting wheel
        Using cached wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
      Collecting cython
        Using cached Cython-3.0.4-cp312-cp312-win_amd64.whl.metadata (3.2 kB)
      Collecting pyarrow<10.1.0,>=10.0.1
        Using cached pyarrow-10.0.1.tar.gz (994 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting numpy>=1.16.6 (from pyarrow<10.1.0,>=10.0.1)
        Using cached numpy-1.26.1-cp312-cp312-win_amd64.whl.metadata (61 kB)
      Using cached setuptools-68.2.2-py3-none-any.whl (807 kB)
      Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
      Using cached Cython-3.0.4-cp312-cp312-win_amd64.whl (2.8 MB)
      Using cached numpy-1.26.1-cp312-cp312-win_amd64.whl (15.5 MB)
      Building wheels for collected packages: pyarrow
        Building wheel for pyarrow (pyproject.toml): started
        Building wheel for pyarrow (pyproject.toml): finished with status 'error'
        error: subprocess-exited-with-error

        Building wheel for pyarrow (pyproject.toml) did not run successfully.
        exit code: 1

        [290 lines of output]
        Could not find platform independent libraries <prefix>
        <string>:36: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
        WARNING setuptools_scm.pyproject_reading toml section missing 'pyproject.toml does not contain a tool.setuptools_scm section'
        running bdist_wheel
        running build
        running build_py
        creating build
        creating build\lib.win-amd64-cpython-312
        creating build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\benchmark.py -> build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\cffi.py -> build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\compute.py -> build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\conftest.py -> build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\csv.py -> build\lib.win-amd64-cpython-312\pyarrow

[had to remove similar rows to fit question]

        copying pyarrow\tests\test_util.py -> build\lib.win-amd64-cpython-312\pyarrow\tests
        copying pyarrow\tests\util.py -> build\lib.win-amd64-cpython-312\pyarrow\tests
        copying pyarrow\tests\__init__.py -> build\lib.win-amd64-cpython-312\pyarrow\tests
        creating build\lib.win-amd64-cpython-312\pyarrow\vendored
        copying pyarrow\vendored\docscrape.py -> build\lib.win-amd64-cpython-312\pyarrow\vendored
        copying pyarrow\vendored\version.py -> build\lib.win-amd64-cpython-312\pyarrow\vendored
        copying pyarrow\vendored\__init__.py -> build\lib.win-amd64-cpython-312\pyarrow\vendored
        creating build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\common.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\conftest.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\encryption.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_basic.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_compliant_nested_type.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_dataset.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_data_types.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_datetime.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_encryption.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_metadata.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_pandas.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_parquet_file.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\test_parquet_writer.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        copying pyarrow\tests\parquet\__init__.py -> build\lib.win-amd64-cpython-312\pyarrow\tests\parquet
        running egg_info
        writing pyarrow.egg-info\PKG-INFO
        writing dependency_links to pyarrow.egg-info\dependency_links.txt
        writing entry points to pyarrow.egg-info\entry_points.txt
        writing requirements to pyarrow.egg-info\requires.txt
        writing top-level names to pyarrow.egg-info\top_level.txt
        ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
        reading manifest file 'pyarrow.egg-info\SOURCES.txt'
        reading manifest template 'MANIFEST.in'
        warning: no files found matching '..\LICENSE.txt'
        warning: no files found matching '..\NOTICE.txt'
        warning: no previously-included files matching '*.so' found anywhere in distribution
        warning: no previously-included files matching '*.pyc' found anywhere in distribution
        warning: no previously-included files matching '*~' found anywhere in distribution
        warning: no previously-included files matching '#*' found anywhere in distribution
        warning: no previously-included files matching '.git*' found anywhere in distribution
        warning: no previously-included files matching '.DS_Store' found anywhere in distribution
        no previously-included directories found matching '.asv'
        writing manifest file 'pyarrow.egg-info\SOURCES.txt'
        copying pyarrow\__init__.pxd -> build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\_compute.pxd -> build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\_compute.pyx -> build\lib.win-amd64-cpython-312\pyarrow
        copying pyarrow\_csv.pxd -> build\lib.win-amd64-cpython-312\pyarrow

[had to remove similar rows to fit question]

        copying pyarrow\includes\libarrow_flight.pxd -> build\lib.win-amd64-cpython-312\pyarrow\includes
        copying pyarrow\includes\libarrow_fs.pxd -> build\lib.win-amd64-cpython-312\pyarrow\includes
        copying pyarrow\includes\libarrow_python.pxd -> build\lib.win-amd64-cpython-312\pyarrow\includes
        copying pyarrow\includes\libarrow_substrait.pxd -> build\lib.win-amd64-cpython-312\pyarrow\includes
        copying pyarrow\includes\libgandiva.pxd -> build\lib.win-amd64-cpython-312\pyarrow\includes
        copying pyarrow\includes\libplasma.pxd -> build\lib.win-amd64-cpython-312\pyarrow\includes
        copying pyarrow\includes\__init__.pxd -> build\lib.win-amd64-cpython-312\pyarrow\includes
        creating build\lib.win-amd64-cpython-312\pyarrow\src
        copying pyarrow\src\ArrowPythonConfig.cmake.in -> build\lib.win-amd64-cpython-312\pyarrow\src
        copying pyarrow\src\ArrowPythonFlightConfig.cmake.in -> build\lib.win-amd64-cpython-312\pyarrow\src
        copying pyarrow\src\CMakeLists.txt -> build\lib.win-amd64-cpython-312\pyarrow\src
        copying pyarrow\src\arrow-python-flight.pc.in -> build\lib.win-amd64-cpython-312\pyarrow\src
        copying pyarrow\src\arrow-python.pc.in -> build\lib.win-amd64-cpython-312\pyarrow\src
        creating build\lib.win-amd64-cpython-312\pyarrow\tensorflow
        copying pyarrow\tensorflow\plasma_op.cc -> build\lib.win-amd64-cpython-312\pyarrow\tensorflow
        copying pyarrow\tests\bound_function_visit_strings.pyx -> build\lib.win-amd64-cpython-312\pyarrow\tests
        copying pyarrow\tests\pyarrow_cython_example.pyx -> build\lib.win-amd64-cpython-312\pyarrow\tests
        creating build\lib.win-amd64-cpython-312\pyarrow\src\arrow
        creating build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\CMakeLists.txt -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\api.h -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\arrow_to_pandas.cc -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\arrow_to_pandas.h -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\arrow_to_python_internal.h -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\benchmark.cc -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python

[had to remove similar rows to fit question]

        copying pyarrow\src\arrow\python\serialize.h -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\type_traits.h -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\udf.cc -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\udf.h -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        copying pyarrow\src\arrow\python\visibility.h -> build\lib.win-amd64-cpython-312\pyarrow\src\arrow\python
        creating build\lib.win-amd64-cpython-312\pyarrow\tests\data
        creating build\lib.win-amd64-cpython-312\pyarrow\tests\data\feather
        copying pyarrow\tests\data\feather\v0.17.0.version.2-compression.lz4.feather -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\feather
        creating build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\README.md -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\TestOrcFile.emptyFile.jsn.gz -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\TestOrcFile.emptyFile.orc -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\TestOrcFile.test1.jsn.gz -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\TestOrcFile.test1.orc -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\TestOrcFile.testDate1900.jsn.gz -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\TestOrcFile.testDate1900.orc -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\decimal.jsn.gz -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        copying pyarrow\tests\data\orc\decimal.orc -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\orc
        creating build\lib.win-amd64-cpython-312\pyarrow\tests\data\parquet
        copying pyarrow\tests\data\parquet\v0.7.1.all-named-index.parquet -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\parquet
        copying pyarrow\tests\data\parquet\v0.7.1.column-metadata-handling.parquet -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\parquet
        copying pyarrow\tests\data\parquet\v0.7.1.parquet -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\parquet
        copying pyarrow\tests\data\parquet\v0.7.1.some-named-index.parquet -> build\lib.win-amd64-cpython-312\pyarrow\tests\data\parquet
        running build_ext
        creating C:\Users\myusername\AppData\Local\Temp\pip-install-3n6isyev\pyarrow_2f98fb6170b843699540aba751d4b9e7\build\cpp
        -- Running CMake for PyArrow C++
        cmake -DARROW_BUILD_DIR=build -DCMAKE_BUILD_TYPE=release -DCMAKE_INSTALL_LIBDIR=lib -DCMAKE_INSTALL_PREFIX=C:\Users\myusername\AppData\Local\Temp\pip-install-3n6isyev\pyarrow_2f98fb6170b843699540aba751d4b9e7\build\dist -DPYTHON_EXECUTABLE=C:\Users\myusername\AppData\Local\Programs\Python\Python312\python.exe -DPython3_EXECUTABLE=C:\Users\myusername\AppData\Local\Programs\Python\Python312\python.exe -DPYARROW_CXXFLAGS= -DPYARROW_WITH_DATASET=off -DPYARROW_WITH_PARQUET_ENCRYPTION=off -DPYARROW_WITH_HDFS=off -DPYARROW_WITH_FLIGHT=off -G "Visual Studio 15 2017 Win64" C:\Users\myusername\AppData\Local\Temp\pip-install-3n6isyev\pyarrow_2f98fb6170b843699540aba751d4b9e7\pyarrow/src
        error: command 'cmake' failed: None
        [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
        ERROR: Failed building wheel for pyarrow
      Failed to build pyarrow
      ERROR: Could not build wheels for pyarrow, which is required to install pyproject.toml-based projects
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Solution

  • You are getting this error because pyarrow still does not support python 3.12.

    There are no wheels yet for 3.12 on pypi

    https://pypi.org/project/pyarrow/#files

    Here is the complete discussion :

    https://github.com/apache/arrow/issues/37880

    A lot of packages are dependent on pyarraow.

    3 weeks ago they posted:

    Due to the current complexity on the release process and the Apache guidelines unfortunately they won't be available just yet but I'll work on creating the pyarrow wheels for Python 3.12 and will add it to the 14.0.0 release.

    Last week they posted:

    I have merged the wheels support for 3.12. Those will be available as nightly development in our next daily builds. The PyPI availability will be ready as soon as we release Arrow 14.0.0 which we started today with preparations but takes some time (~2 weeks)

    Hopefully, it will be available soon.

    Till then you can use python 3.11 for snowflake-connector