pythonwindowscmakeinstallationcaffe
Script cannot find python libraries and numpy when I try to install caffe on Windows 10
I have been using this resource to install caffe on my Windows 10 system.
I started by installing the necessary dependencies for Python via the command prompt:
C:\Users\MYNAME>conda config --add channels conda-forge
C:\Users\MYNAME>conda config --add channels willyd
C:\Users\MYNAME>conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz
I then created an empty directory called caffe in my C:\ directory. I then went into that directory via the command prompt and typed the following commands:
C:\caffe\caffe>git clone https://github.com/BVLC/caffe.git
C:\caffe\caffe>cd caffe
I then opened the file build_win.cmd in the /scripts directory and edited a few things. My file after editing looks like this:
@echo off
@setlocal EnableDelayedExpansion
:: Default values
if DEFINED APPVEYOR (
echo Setting Appveyor defaults
if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14
if NOT DEFINED WITH_NINJA set WITH_NINJA=0
if NOT DEFINED CPU_ONLY set CPU_ONLY=1
if NOT DEFINED CUDA_ARCH_NAME set CUDA_ARCH_NAME=Auto
if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release
if NOT DEFINED USE_NCCL set USE_NCCL=0
if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0
if NOT DEFINED PYTHON_VERSION set PYTHON_VERSION=3
if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1
if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1
if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0
if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python
if NOT DEFINED RUN_TESTS set RUN_TESTS=1
if NOT DEFINED RUN_LINT set RUN_LINT=1
if NOT DEFINED RUN_INSTALL set RUN_INSTALL=1
:: Set python 2.7 with conda as the default python
if !PYTHON_VERSION! EQU 2 (
**set CONDA_ROOT=C:\Users\MYNAME\Anaconda3**
)
:: Set python 3.5 with conda as the default python
if !PYTHON_VERSION! EQU 3 (
**set CONDA_ROOT=C:\Users\MYNAME\Anaconda3**
)
**set PATH=C:\Users\MYNAME\Anaconda3;C:\Users\MYNAME\Anaconda3\Scripts;C:\Users\MYNAME\Anaconda3\Library\bin;!PATH!**
:: Check that we have the right python version
!PYTHON_EXE! --version
:: Add the required channels
conda config --add channels conda-forge
conda config --add channels willyd
:: Update conda
conda update conda -y
:: Download other required packages
conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz
if ERRORLEVEL 1 (
echo ERROR: Conda update or install failed
exit /b 1
)
:: Install cuda and disable tests if needed
if !WITH_CUDA! == 1 (
call %~dp0\appveyor\appveyor_install_cuda.cmd
set CPU_ONLY=0
set RUN_TESTS=0
set USE_NCCL=1
) else (
set CPU_ONLY=1
)
:: Disable the tests in debug config
if "%CMAKE_CONFIG%" == "Debug" (
echo Disabling tests on appveyor with config == %CMAKE_CONFIG%
set RUN_TESTS=0
)
:: Disable linting with python 3 until we find why the script fails
if !PYTHON_VERSION! EQU 3 (
set RUN_LINT=0
)
) else (
:: Change the settings here to match your setup
:: Change MSVC_VERSION to 12 to use VS 2013
if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14
:: Change to 1 to use Ninja generator (builds much faster)
if NOT DEFINED WITH_NINJA set WITH_NINJA=0
:: Change to 1 to build caffe without CUDA support
if NOT DEFINED CPU_ONLY set CPU_ONLY=1
:: Change to generate CUDA code for one of the following GPU architectures
:: [Fermi Kepler Maxwell Pascal All]
if NOT DEFINED CUDA_ARCH_NAME set CUDA_ARCH_NAME=Auto
:: Change to Debug to build Debug. This is only relevant for the Ninja generator the Visual Studio generator will generate both Debug and Release configs
if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release
:: Set to 1 to use NCCL
if NOT DEFINED USE_NCCL set USE_NCCL=0
:: Change to 1 to build a caffe.dll
if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0
:: Change to 3 if using python 3.5 (only 2.7 and 3.5 are supported)
if NOT DEFINED PYTHON_VERSION set PYTHON_VERSION=3
:: Change these options for your needs.
if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1
if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1
if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0
:: If python is on your path leave this alone
if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python
:: Run the tests
if NOT DEFINED RUN_TESTS set RUN_TESTS=0
:: Run lint
if NOT DEFINED RUN_LINT set RUN_LINT=0
:: Build the install target
if NOT DEFINED RUN_INSTALL set RUN_INSTALL=0
)
:: Set the appropriate CMake generator
:: Use the exclamation mark ! below to delay the
:: expansion of CMAKE_GENERATOR
if %WITH_NINJA% EQU 0 (
if "%MSVC_VERSION%"=="14" (
set CMAKE_GENERATOR=Visual Studio 14 2015 Win64
)
if "%MSVC_VERSION%"=="12" (
set CMAKE_GENERATOR=Visual Studio 12 2013 Win64
)
if "!CMAKE_GENERATOR!"=="" (
echo ERROR: Unsupported MSVC version
exit /B 1
)
) else (
set CMAKE_GENERATOR=Ninja
)
echo INFO: ============================================================
echo INFO: Summary:
echo INFO: ============================================================
echo INFO: MSVC_VERSION = !MSVC_VERSION!
echo INFO: WITH_NINJA = !WITH_NINJA!
echo INFO: CMAKE_GENERATOR = "!CMAKE_GENERATOR!"
echo INFO: CPU_ONLY = !CPU_ONLY!
echo INFO: CUDA_ARCH_NAME = !CUDA_ARCH_NAME!
echo INFO: CMAKE_CONFIG = !CMAKE_CONFIG!
echo INFO: USE_NCCL = !USE_NCCL!
echo INFO: CMAKE_BUILD_SHARED_LIBS = !CMAKE_BUILD_SHARED_LIBS!
echo INFO: PYTHON_VERSION = !PYTHON_VERSION!
echo INFO: BUILD_PYTHON = !BUILD_PYTHON!
echo INFO: BUILD_PYTHON_LAYER = !BUILD_PYTHON_LAYER!
echo INFO: BUILD_MATLAB = !BUILD_MATLAB!
echo INFO: PYTHON_EXE = "!PYTHON_EXE!"
echo INFO: RUN_TESTS = !RUN_TESTS!
echo INFO: RUN_LINT = !RUN_LINT!
echo INFO: RUN_INSTALL = !RUN_INSTALL!
echo INFO: ============================================================
:: Build and exectute the tests
:: Do not run the tests with shared library
if !RUN_TESTS! EQU 1 (
if %CMAKE_BUILD_SHARED_LIBS% EQU 1 (
echo WARNING: Disabling tests with shared library build
set RUN_TESTS=0
)
)
if NOT EXIST build mkdir build
pushd build
:: Setup the environement for VS x64
set batch_file=!VS%MSVC_VERSION%0COMNTOOLS!..\..\VC\vcvarsall.bat
call "%batch_file%" amd64
:: Configure using cmake and using the caffe-builder dependencies
:: Add -DCUDNN_ROOT=C:/Projects/caffe/cudnn-8.0-windows10-x64-v5.1/cuda ^
:: below to use cuDNN
cmake -G"!CMAKE_GENERATOR!" ^
-DBLAS=Open ^
-DCMAKE_BUILD_TYPE:STRING=%CMAKE_CONFIG% ^
-DBUILD_SHARED_LIBS:BOOL=%CMAKE_BUILD_SHARED_LIBS% ^
-DBUILD_python:BOOL=%BUILD_PYTHON% ^
-DBUILD_python_layer:BOOL=%BUILD_PYTHON_LAYER% ^
-DBUILD_matlab:BOOL=%BUILD_MATLAB% ^
-DCPU_ONLY:BOOL=%CPU_ONLY% ^
-DCOPY_PREREQUISITES:BOOL=1 ^
-DINSTALL_PREREQUISITES:BOOL=1 ^
-DUSE_NCCL:BOOL=!USE_NCCL! ^
-DCUDA_ARCH_NAME:STRING=%CUDA_ARCH_NAME% ^
"%~dp0\.."
if ERRORLEVEL 1 (
echo ERROR: Configure failed
exit /b 1
)
:: Lint
if %RUN_LINT% EQU 1 (
cmake --build . --target lint --config %CMAKE_CONFIG%
)
if ERRORLEVEL 1 (
echo ERROR: Lint failed
exit /b 1
)
:: Build the library and tools
cmake --build . --config %CMAKE_CONFIG%
if ERRORLEVEL 1 (
echo ERROR: Build failed
exit /b 1
)
:: Build and exectute the tests
if !RUN_TESTS! EQU 1 (
cmake --build . --target runtest --config %CMAKE_CONFIG%
if ERRORLEVEL 1 (
echo ERROR: Tests failed
exit /b 1
)
if %BUILD_PYTHON% EQU 1 (
if %BUILD_PYTHON_LAYER% EQU 1 (
:: Run python tests only in Release build since
:: the _caffe module is _caffe-d is debug
if "%CMAKE_CONFIG%"=="Release" (
:: Run the python tests
cmake --build . --target pytest
if ERRORLEVEL 1 (
echo ERROR: Python tests failed
exit /b 1
)
)
)
)
)
if %RUN_INSTALL% EQU 1 (
cmake --build . --target install --config %CMAKE_CONFIG%
)
popd
@endlocal
After this, I ran the following command:
C:\caffe\caffe>scripts\build_win.cmd
This gave me the following output:
The system cannot find the drive specified.
The system cannot find the drive specified.
INFO: ============================================================
INFO: Summary:
INFO: ============================================================
INFO: MSVC_VERSION = 14
INFO: WITH_NINJA = 0
INFO: CMAKE_GENERATOR = "Visual Studio 14 2015 Win64"
INFO: CPU_ONLY = 0
INFO: CUDA_ARCH_NAME = Auto
INFO: CMAKE_CONFIG = Release
INFO: USE_NCCL = 0
INFO: CMAKE_BUILD_SHARED_LIBS = 0
INFO: PYTHON_VERSION = 2
INFO: BUILD_PYTHON = 1
INFO: BUILD_PYTHON_LAYER = 1
INFO: BUILD_MATLAB = 0
INFO: PYTHON_EXE = "python"
INFO: RUN_TESTS = 0
INFO: RUN_LINT = 0
INFO: RUN_INSTALL = 0
INFO: ============================================================
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler using: Visual Studio 14 2015 Win64
-- Check for working C compiler using: Visual Studio 14 2015 Win64 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler using: Visual Studio 14 2015 Win64
-- Check for working CXX compiler using: Visual Studio 14 2015 Win64 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: C:/cygwin64/bin/python2.7.exe (found suitable version "2.7.14", minimum required is "2.7")
-- Downloading prebuilt dependencies to C:/Users/Abdullah Siddiqui/.caffe/dependencies/download/libraries_v140_x64_py27_1.1.0.tar.bz2
-- [download 0% complete]
...
-- Extracting dependencies
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Boost version: 1.61.0
-- Found the following Boost libraries:
-- system
-- thread
-- filesystem
-- chrono
-- date_time
-- atomic
-- Found GFlags: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found gflags (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: gflags_shared)
-- Found Glog: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found glog (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: glog)
-- Found Protobuf: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/bin/protoc.exe (found version "3.1.0")
-- Found PROTOBUF Compiler: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/bin/protoc.exe
-- Found LMDB: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found lmdb (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: lmdb)
-- Found LevelDB: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found LevelDB (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: leveldb)
-- Found ZLIB: optimized;C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlib.lib;debug;C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlibd.lib (found version "1.2.8")
-- Found Snappy: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found Snappy (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: snappy_static;optimized;C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlib.lib;debug;C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlibd.lib)
CMake Warning at cmake/Dependencies.cmake:97 (message):
-- CUDA is not detected by cmake. Building without it...
Call Stack (most recent call first):
CMakeLists.txt:80 (include)
-- OpenCV found (C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries)
-- Found OpenBLAS libraries: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/libopenblas.dll.a
-- Found OpenBLAS include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "2.7")
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named numpy
-- Could NOT find NumPy (missing: NUMPY_INCLUDE_DIR NUMPY_VERSION) (Required is at least version "1.7.1")
-- Boost version: 1.61.0
-- Found the following Boost libraries:
-- python
-- Python interface is disabled or not all required dependencies found. Building without it...
-- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.10.1.windows.1")
--
-- ******************* Caffe Configuration Summary *******************
-- General:
-- Version : 1.0.0
-- Git : 1.0-111-g509dae0-dirty
-- System : Windows
-- C++ compiler : C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
-- Release CXX flags : /MD /O2 /Ob2 /D NDEBUG /DWIN32 /D_WINDOWS /W3 /GR /EHsc
-- Debug CXX flags : /D_DEBUG /MDd /Zi /Ob0 /Od /RTC1 /DWIN32 /D_WINDOWS /W3 /GR /EHsc
-- Build type : Release
--
-- BUILD_SHARED_LIBS : 0
-- BUILD_python : 1
-- BUILD_matlab : 0
-- BUILD_docs :
-- CPU_ONLY : 0
-- USE_OPENCV : ON
-- USE_LEVELDB : ON
-- USE_LMDB : ON
-- USE_NCCL : 0
-- ALLOW_LMDB_NOLOCK : OFF
--
-- Dependencies:
-- BLAS : Yes (Open)
-- Boost : Yes (ver. 1.61)
-- glog : Yes
-- gflags : Yes
-- protobuf : Yes (ver. 3.1.0)
-- lmdb : Yes (ver. 0.9.70)
-- LevelDB : Yes (ver. 1.18)
-- Snappy : Yes (ver. 1.1.1)
-- OpenCV : Yes (ver. 3.1.0)
-- CUDA : No
--
-- Install:
-- Install path : C:/Users/Abdullah Siddiqui/caffe/build/install
--
-- Configuring done
CMake Error at CMakeLists.txt:143 (add_dependencies):
The dependency target "pycaffe" of target "pytest" does not exist.
-- Generating done
CMake Warning:
Manually-specified variables were not used by the project:
BUILD_python_layer
CUDA_ARCH_NAME
-- Build files have been written to: C:/Users/Abdullah Siddiqui/caffe/build
ERROR: Configure failed
I am particularly puzzled by these statements:
-- Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "2.7")
Traceback (most recent call last):
File "<string>", line 1, in <module>
ImportError: No module named numpy
-- Could NOT find NumPy (missing: NUMPY_INCLUDE_DIR NUMPY_VERSION) (Required is at least version "1.7.1")
How are these files not being detected in my system when build_win.cmd clearly contains this line:
set PATH=C:\Users\MYNAME\Anaconda3;C:\Users\ MYNAME\Anaconda3\Scripts;C:\Users\MYNAME\Anaconda3\Library\bin;!PATH!
The relevant python files are also present in my system.
PythonLibs, which the .cmd file is unable to find, is present in the Anaconda3 directory.
Numpy is also present in one of the sub-directories of that directory.
I can even run python directly from the command line:
At this point, I have no I idea why I am getting errors about numpy and Python libraries. Please let me know if you have any suggestions.
EDIT:
I ran the following command in my project directory after watching this video:
C:\caffe\caffe>python scripts\download_prebuilt_dependencies.py --msvc_version=v140
The output was as follows:
ERROR: Could not find url for MSVC version = v140 and Python version = 3.6.
Available combinations are: [('v120', '2.7'), ('v140', '2.7'), ('v140', '3.5')]
Does this mean I'll have to switch to Python 3.5 to use this framework?
EDIT:
I downloaded python 3.5.5, performed the same steps as above and got the same errors.
I thought about CristiFati's suggestion more and executed the following command:
C:\caffe\caffe\build>cmake -DPYTHON_EXECUTABLE=C:\Users\Abdullah Siddiqui\Anaconda3\python.exe ..
This gave me the following output:
-- Building for: Visual Studio 14 2015
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler using: Visual Studio 14 2015
-- Check for working C compiler using: Visual Studio 14 2015 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler using: Visual Studio 14 2015
-- Check for working CXX compiler using: Visual Studio 14 2015 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find PythonInterp: Found unsuitable version "1.4", but required is at least "2.7" (found C:/Users/Abdullah)
CMake Error at cmake/WindowsDownloadPrebuiltDependencies.cmake:40 (message):
Could not find url for MSVC version = 1900 and Python version = 1.4.
Call Stack (most recent call first):
CMakeLists.txt:77 (include)
-- Configuring incomplete, errors occurred!
See also "C:/Users/.../projects/caffe/build/CMakeFiles/CMakeOutput.log".
I now understand that the problem is being caused by the executable of Python 2.7 being used for compilation. How can I make the build script point to python 3.5?
EDIT:
I ran the cmake command correctly as shown below but I am now seeing new errors.
C:\Users\MYNAME\projects\caffe\build>cmake -DPYTHON_EXECUTABLE="C:\Users\MYNAME\Anaconda3\python.exe" ..
It is giving me these errors:
-- Could NOT find Glog (missing: GLOG_LIBRARY)
CMake Error at cmake/ProtoBuf.cmake:6 (find_package):
Could not find a configuration file for package "Protobuf" that is
compatible with requested version "".
Solution
cmake has "standard" ways locate external packages installations (via scripts). This applies to Python. When such a package is found there are some variables that are set, typically:
${PACKAGE}_FOUND=ON(package status flag)${PACKAGE}_INCLUDE_DIRS(include directories - used at compile time)${PACKAGE}_LIBRARIES(libraries - used at link time)
When searching for numpy ([GitHub]: BVLC/caffe - (windows) caffe/cmake/Modules/FindNumPy.cmake), PythonInterp (standard cmake module) is searched (and used). As noticed in the initial output:
-- Found PythonInterp: C:/cygwin64/bin/python2.7.exe (found suitable version "2.7.14", minimum required is "2.7")
python 2.7 (which doesn't have the numpy module installed) was detected, leading to numpy finding failure.
Looking at FindPythonInterp.cmake script I noticed that it sets (among others) the ${PYTHON_EXECUTABLE} (cmake) variable.
Setting that variable "manually" to cmake (by passing a cmdline argument), would be the next logical thing (as I didn't know if this alone was enough, I just added a comment):
-DPYTHON_EXECUTABLE="c:\path\to\python3"
Note: Do not mix Cygwin and native Win tools, as you might run into subtle problems.