Search code examples
pythoncudatensorflowgpucudnn

cuda kernel doesn't match during installation of Tensorflow

I am installing Tensorflow as instructed on the official page and at the step of "Validate your installation".

enter image description here

>>> sess = tf.Session()
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
E tensorflow/stream_executor/cuda/cuda_driver.cc:509] failed call to cuInit: CUDA_ERROR_NO_DEVICE
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: Shu
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: Shu
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 375.26.0
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:363] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  367.57  Mon Oct  3 20:37:01 PDT 2016
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 
"""
I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 367.57.0
E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version 367.57.0 does not match DSO version 375.26.0 -- cannot find working devices in this configuration

This is what I got when I try to start a session. I don't quite understand what's going on. Please kindly help. Thank you!

Solution

  • I don't believe the error is related to Tensorflow. You should get the same error running nvidia-smi.

    Is it possible you have updated your NVIDIA GPU drivers after installing the CUDA toolkit? It looks like the toolkit expects drivers version 367.57, while you are running a more recent version, 375.26.

    For a quick check, try reverting to NVIDIA drivers version 367.57; you can do it from System Settings > Software and Updates > Additional Drivers.

    Once confirmed the mismatch in expected drivers version is the issue, you can either stay with drivers version 367.57, or un-install CUDA Toolkit and cuDNN, update the drivers to 375.26, and the re-install CUDA Toolkit and cuDNN.