What should I link against: The actual CUDA driver library or the driver library stub?

Question

A CUDA distribution, at least on Linux, has a "stub libraries" directory, which contains among others a libcuda.so file - named the same as an actual NVIDIA driver library.

When build a CUDA program which makes driver API calls, on a system with both CUDA and the CUDA driver installed - when should I be linking against the driver library stub, and when should I link against the actual driver library?

Notes:

• In case it matters, assume a recent CUDA release (version 11.x or 12.x).
• The NVIDIA driver might be different than the driver version bundled with the CUDA distribution, i.e. that may be one of the factors.
• If we're using a build system generator, e.g. CMake, this question is basically moot, since you let it decide where to locate the relevant libraries - and its choice works. See this question about doing that.

Answer 1

I would always counsel to link against the stub library rather than the local libcuda, where one exists. I say that for exactly three reasons:

1. It’s portable. If you provision any system with the toolkit, you have the stub library which matches the toolkit version you have, whether it is on a machine with GPU hardware or not.
2. It’s in a predictable location. If you know where the toolkit is, you (or just about any build system) can derive a relative path that will get you to the stub. That isn’t the case for the local libcuda library, particularly on systems which provide third party repackaged drivers, where things can be in very non-standard places. I have seen this wreck havoc on automatic packaging and distribution systems.
3. It is what nvcc does. When in Rome….