I am attempting to install clickhouse-cityhash
with pip
on OSX 10.14.5 but it fails with the following (abridged) output:
src/city.cc:396:5: error: use of undeclared identifier '_mm_crc32_u64'
CHUNK(1, 1); CHUNK(k0, 0);
^
...
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: command 'cc' failed with exit status 1
I've also tried compiling via CC=gcc
and CC=g++
to no avail.
The command that is run on failure is:
cc -fno-strict-aliasing -fno-common \
-dynamic -g -Os -pipe -fno-common \
-fno-strict-aliasing -fwrapv -DENABLE_DTRACE \
-DMACOSX -DNDEBUG -Wall -Wstrict-prototypes \
-Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall \
-Wstrict-prototypes -DENABLE_DTRACE -arch i386 \
-arch x86_64 -pipe -Iinclude \
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 \
-c src/city.cc -o build/temp.macosx-10.14-intel-2.7/src/city.o \
-O3 -msse4.2 -Wno-unused-value -Wno-unused-function
In my attempt to understand the problem, I looked at the source code and I can see four calls to _mm_crc32_u64
that comprise part of the CHUNK
preprocessor directive mentioned in the error log:
f = _mm_crc32_u64(f, a); \
g = _mm_crc32_u64(g, b); \
h = _mm_crc32_u64(h, c); \
i = _mm_crc32_u64(i, d); \
j = _mm_crc32_u64(j, e); \
I found a reference to _mm_crc32_u64
in the Intel Intrinsics Guide so my understanding is that it's an Intel Intrinsic Instruction as a C function that's part of the SSE4.2 instruction set.
I figured that my machine does not include the SSE4.2 instruction set, but when I run the following command:
sysctl -a | grep cpu.features
SSE4.2 is included in the list:
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
Therefore, should I expect _mm_crc32_u64
to be available, and if so, what is the likely reason for this error?
If not, is there anything I can do to make these instructions available?
Many thanks to @PeterCordes for his very valuable observations in the question comments above!
The failing build command during pip install clickhouse-cityhash
included the -arch i386
flag. The default behaviour of x86 clang is to build 64-bit code despite the presence of this flag.
However, this does not appear to be Apple clang's default behaviour. If 32-bit code is generated then _mm_crc32_u32
would the largest CRC available, implying that _mm_crc32_u64
is not defined.
Therefore, one solution is not to use Apple clang.
Most developers using OSX will be familiar with the brew
package manager and have it installed. You may find that you already have a version of gcc
installed via brew
as a dependency of another package.
Check with the following:
brew list | grep gcc
If not, install it with:
brew install gcc
The executable should be available in your $PATH
(usually at /usr/local/bin
) as gcc
or similar—mine was available as gcc-8
.
To use, just define the gcc
you want to use with the CC
envvar and run pip install
; e.g:
CC=gcc-8 pip install clickhouse-cityhash
Hope this helps :)