How to prevent NVIDIA drivers from automatically upgrading on Ubuntu?

I was training models last night on my Ubuntu workstation, and then woke up this morning and saw this message:

Failed to initialize NVML: Driver/library version mismatch

Apparently the NVIDIA system driver automatically updated itself, and now I need to reboot the machine to use my GPUs... How do I prevent automatic updates from NVIDIA?

Solution

If this is because of unattended upgrades (likely), then you should add the NVIDIA drivers to it's blacklist. Add nvidia- and libnvidia- to /etc/apt/apt.conf.d/50unattended-upgrades like so:

Unattended-Upgrade::Package-Blacklist {
        "nvidia-";
        "libnvidia-";
        ...
}

Consider adding additional lines if you see any other NVIDIA driver names in the output of apt list --installed | grep nv