Missing instruction `_mm_crc32_u64` when installing clickhouse-cityhash on OSX 10.14.5

I am attempting to install clickhouse-cityhash with pip on OSX 10.14.5 but it fails with the following (abridged) output:

src/city.cc:396:5: error: use of undeclared identifier '_mm_crc32_u64'
    CHUNK(1, 1); CHUNK(k0, 0);
    ^
...
fatal error: too many errors emitted, stopping now [-ferror-limit=]
  20 errors generated.
  error: command 'cc' failed with exit status 1

I've also tried compiling via CC=gcc and CC=g++ to no avail.

The command that is run on failure is:

cc -fno-strict-aliasing -fno-common \
   -dynamic -g -Os -pipe -fno-common \
   -fno-strict-aliasing -fwrapv -DENABLE_DTRACE \
   -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes \
   -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall \
   -Wstrict-prototypes -DENABLE_DTRACE -arch i386 \
   -arch x86_64 -pipe -Iinclude \
   -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 \
   -c src/city.cc -o build/temp.macosx-10.14-intel-2.7/src/city.o \
   -O3 -msse4.2 -Wno-unused-value -Wno-unused-function

In my attempt to understand the problem, I looked at the source code and I can see four calls to _mm_crc32_u64 that comprise part of the CHUNK preprocessor directive mentioned in the error log:

f = _mm_crc32_u64(f, a);                                    \
g = _mm_crc32_u64(g, b);                                    \
h = _mm_crc32_u64(h, c);                                    \
i = _mm_crc32_u64(i, d);                                    \
j = _mm_crc32_u64(j, e);                                    \

I found a reference to _mm_crc32_u64 in the Intel Intrinsics Guide so my understanding is that it's an Intel Intrinsic Instruction as a C function that's part of the SSE4.2 instruction set.

I figured that my machine does not include the SSE4.2 instruction set, but when I run the following command:

sysctl -a | grep cpu.features

SSE4.2 is included in the list:

machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX SMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C

Therefore, should I expect _mm_crc32_u64 to be available, and if so, what is the likely reason for this error?

If not, is there anything I can do to make these instructions available?

Solution

Many thanks to @PeterCordes for his very valuable observations in the question comments above!

The failing build command during pip install clickhouse-cityhash included the -arch i386 flag. The default behaviour of x86 clang is to build 64-bit code despite the presence of this flag.

However, this does not appear to be Apple clang's default behaviour. If 32-bit code is generated then _mm_crc32_u32 would the largest CRC available, implying that _mm_crc32_u64 is not defined.

Therefore, one solution is not to use Apple clang.

Most developers using OSX will be familiar with the brew package manager and have it installed. You may find that you already have a version of gcc installed via brew as a dependency of another package.

Check with the following:

brew list | grep gcc

If not, install it with:

brew install gcc

The executable should be available in your $PATH (usually at /usr/local/bin) as gcc or similar—mine was available as gcc-8.

To use, just define the gcc you want to use with the CC envvar and run pip install; e.g:

CC=gcc-8 pip install clickhouse-cityhash

Hope this helps :)