I tried to compile Tensorflow myself to speed up the training (using a precompiled wheel works but is slow). I use ./configure to configure and explicitly specify that cuDNN version 5.1.10 should be used. Before that, I downloaded cuDNN 5.1.10 and copied my files to the Cuda directory.
Here is what the config (.tf_configure.bazelrc) looks like:
build --action_env PYTHON_BIN_PATH="/home/ubuntu/project/venv/bin/python"
build --action_env PYTHON_LIB_PATH="/home/ubuntu/project/venv/lib/python3.5/site-packages"
build --define PYTHON_BIN_PATH="/home/ubuntu/project/venv/bin/python"
build --define PYTHON_LIB_PATH="/home/ubuntu/project/venv/lib/python3.5/site-packages"
build --force_python=py3
build --host_force_python=py3
build --python3_path="/home/ubuntu/project/venv/bin/python"
test --force_python=py3
test --host_force_python=py3
test --define PYTHON_BIN_PATH="/home/ubuntu/project/venv/bin/python"
test --define PYTHON_LIB_PATH="/home/ubuntu/project/venv/lib/python3.5/site-packages"
run --define PYTHON_BIN_PATH="/home/ubuntu/project/venv/bin/python"
run --define PYTHON_LIB_PATH="/home/ubuntu/project/venv/lib/python3.5/site-packages"
build --define with_jemalloc=true
build:opt --cxxopt=-march=native --copt=-march=native
build --action_env TF_NEED_CUDA="1"
build --action_env TF_NEED_OPENCL="0"
build --action_env TF_CUDA_CLANG="0"
build --action_env CUDA_TOOLKIT_PATH="/usr/local/cuda"
build --action_env TF_CUDA_VERSION="8.0"
build --action_env GCC_HOST_COMPILER_PATH="/usr/bin/gcc"
build --action_env TF_CUDNN_VERSION="5.1.10"
build --action_env CUDNN_INSTALL_PATH="/usr/local/cuda-8.0"
build --action_env TF_CUDNN_VERSION="5.1.10"
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="3.7"
build --config=cuda
test --config=cuda
Notice the TF_CUDNN_VERSION="5.1.10" part. Next I execute the following commands:
bazel build --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
pip3 install --upgrade /tmp/tensorflow_pkg/tensorflow-1.2.0-cp35-cp35m-linux_x86_64.whl
And finally, I get the following error message
Loaded runtime CuDNN library: 5110 (compatibility version 5100) but source was compiled with 5005 (compatibility version 5000). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.
What is going on? Somehow Tensorflow is compiled using cuDNN 5.0 even though I specified the correct version?!
The system is Ubuntu 16, AWS p2 instance.
The only way I could fix this was to start all over again (a fresh version of Ubuntu from the AWS marketplace).
I will accept this answer since there is no point in going back and trying to fix the problem but in case somebody else is wondering about this, I found this article quite helpful: https://developer.nvidia.com/compute/cuda/8.0/Prod2/local_installers/cuda_8.0.61_375.26_linux-run