I have installed caffe in a Python 3.5 anaconda environment using the following command:
conda install -c anaconda caffe-gpu. The machine is a Ubuntu machine with CUDA already properly configured. The installation happened without errors and I can import caffe without issues.
I am trying to run pascal_multilabel_with_data_layer.ipynb jupyter notebook example provided with Caffe (https://github.com/BVLC/caffe/tree/master/examples). I have downloaded the PASCAL VOC dataset and put in the appropriate folder (data folder).
I have made sure caffe is set to gpu mode in the notebook as follows:
caffe.set_mode_gpu()
caffe.set_device(0)
Nvidia SMI does show Caffe occupying space on GPU 0 as expected. I go through all the sections of the notebook without errors. I made sure the solver.prototext file in models folder is set to use GPU as solver_mode: GPU.
I made sure caffe is properly configured by running the following command inside the virtual environment:
python -c 'import caffe; caffe.set_mode_gpu()' 2>/dev/null && echo "Success" || echo "Failure". The command returns success.
However, when I run the training process, it seems that the GPU is unused and CPU usage goes high, which indicates Caffe is not using GPU for computations. Any idea on how to further debug the issue?
Don't know why people give unlikes when they can't answer.
Anyways, there was a clash between system python3 and conda python3. Also conda forge's caffe does not really work. I had to recompile caffe from source and remove conda RWE restrictions to install packages directly in conda base directory. Only then it properly worked.