Numba cuda Possible mix of compiler/IR from different releases

I've been trying to run some numba/cuda code, like this module:

https://github.com/Maghoumi/pytorch-softdtw-cuda/blob/master/soft_dtw_cuda.py

However I run into the following error:

numba.cuda.cudadrv.error.NvvmError: Failed to compile

IR version 1.6 incompatible with current version 2.0
<unnamed>: error: incompatible IR detected. Possible mix of compiler/IR from different releases.
NVVM_ERROR_IR_VERSION_MISMATCH

I guess I installed incompatible versions for some packages, but have no idea where to start. Which packages are concerned?

Solution

The underlying reason for this appears to be using CUDA 12.

According to the CUDA 12 release notes:

NVVM IR Update: with CUDA 12.0 we are releasing NVVM IR 2.0 which is incompatible with NVVM IR 1.x accepted by the libNVVM compiler in prior CUDA toolkit releases. Users of the libNVVM compiler in CUDA 12.0 toolkit must generate NVVM IR 2.0.

From the error, it would appear that the Numba CUDA backend is generating NVVM IR 1.6, and from the release notes for CUDA 12, NVVM IR 1.6 is no longer supported by the NVVM compiler library supplied in CUDA 12.

In the short term, use CUDA 11.x or earlier. In the longer term, report this as a bug to the Numba developers and get them to update their compiler infrastructure to match the CUDA 12 NVVM requirements.