Search code examples
pythonpipapple-m1pyarrowapache-arrow

pyarrow==0.17.1 installation ends with fatal error: 'arrow/python/config.h' file not found (Apple M1 chip)

I'm trying to install the pyarrow==0.17.1 package on my MacBook pro machine (Apple M1 chip) and I'm getting the following error

      -- Found PkgConfig: /opt/homebrew/bin/pkg-config (found version "0.29.2")
      -- Found Arrow: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/include (found version "7.0.0")
      -- Arrow version: 7.0.0 (HOME: /opt/homebrew/Cellar/apache-arrow/7.0.0_3)
      -- Arrow SO and ABI version: 700
      -- Arrow full SO version: 700.0.0
      -- Found the Arrow core shared library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow.dylib
      -- Found the Arrow core import library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow.dylib
      -- Found the Arrow core static library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow.a
      -- Found ArrowPython: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/include (found version "7.0.0")
      -- Found the Arrow Python by HOME: /opt/homebrew/Cellar/apache-arrow/7.0.0_3
      -- Found the Arrow Python shared library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow_python.dylib
      -- Found the Arrow Python import library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow_python.dylib
      -- Found the Arrow Python static library: /opt/homebrew/Cellar/apache-arrow/7.0.0_3/lib/libarrow_python.a
      -- Configuring done
      -- Generating done
      -- Build files have been written to: /private/var/folders/6m/ychzdp6s7l390zfm77myvxzr0000gn/T/pip-install-lv5qrivl/pyarrow_12e0171ed6fa433da71d5392c3e10c3a/build/temp.macosx-12.0-arm64-3.7
      -- Finished cmake for pyarrow
      -- Running cmake --build for pyarrow
      cmake --build . --config release --
      [  6%] Compiling Cython CXX source for lib...
      [  6%] Built target lib_pyx
      [ 13%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
      /private/var/folders/6m/ychzdp6s7l390zfm77myvxzr0000gn/T/pip-install-lv5qrivl/pyarrow_12e0171ed6fa433da71d5392c3e10c3a/build/temp.macosx-12.0-arm64-3.7/lib.cpp:774:10: fatal error: 'arrow/python/config.h' file not found
      #include "arrow/python/config.h"
               ^~~~~~~~~~~~~~~~~~~~~~~
      1 error generated.
      make[2]: *** [CMakeFiles/lib.dir/lib.cpp.o] Error 1
      make[1]: *** [CMakeFiles/lib.dir/all] Error 2
      make: *** [all] Error 2
      error: command 'cmake' failed with exit status 2
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

I couldn't find a proven solution for this problem, the only reference I did review is the following - https://github.com/apache/arrow/issues/2281, as mentioned, no proven solution there.

Versions I'm using are: python 3.7.12 pip 22.0.4

On my machine, I have the following installed - apache-arrow, apache-arrow-glib, cmake. I've also tried to install the package with the --no-use-pep517 flag as the following- pip install --no-use-pep517 pyarrow==0.17.1 - no success there.

I took a peek inside arrow/python/ and config.h is obviously missing, I also look inside arrow/ which does contains a config.h, however, it's not the config.h that was required.

Does anyone have an idea how to solve this issue?

Solution

  • pyarrow==0.17.1 predates M1 chips (May 2020 vs November 2020).

    Any particular reason you wouldn't want to use pyarrow==7.0.0? Your brew installed arrow seems to be 7.0.0 too.