Caffe crashes when I train a net.
In this case I have only used GPU 0. Here is the error trace:
build/tools/caffe train -solver models/mv16f/solver.prototxt -gpu 0
I0113 14:21:05.861361 85242 solver.cpp:289] Learning Rate Policy: step
I0113 14:21:05.862876 85242 solver.cpp:341] Iteration 0, Testing net (#0)
I0113 14:21:30.271030 85242 solver.cpp:409] Test net output #0: accuracy = 0.00872
I0113 14:21:30.271070 85242 solver.cpp:409] Test net output #1: loss = 4.62895 (* 1 = 4.62895 loss)
I0113 14:21:32.317018 85242 solver.cpp:237] Iteration 0, loss = 4.62663
I0113 14:21:32.317062 85242 solver.cpp:253] Train net output #0: loss = 4.62663 (* 1 = 4.62663 loss)
*** Aborted at 1452691298 (unix time) try "date -d @1452691298" if you are using GNU date ***
PC: @ 0x7fe7f65f1cbc caffe::SGDSolver<>::GetLearningRate()
*** SIGFPE (@0x7fe7f65f1cbc) received by PID 85242 (TID 0x7fe7f72057c0) from PID 18446744073548012732; stack trace: ***
@ 0x7fe7f49c0d40 (unknown)
@ 0x7fe7f65f1cbc caffe::SGDSolver<>::GetLearningRate()
@ 0x7fe7f65f2281 caffe::SGDSolver<>::ApplyUpdate()
@ 0x7fe7f65d967c caffe::Solver<>::Step()
@ 0x7fe7f65d8990 caffe::Solver<>::Solve()
@ 0x7fe7f673251e caffe::P2PSync<>::run()
@ 0x416aa6 train()
@ 0x418c9a main
@ 0x7fe7f49abec5 (unknown)
@ 0x415819 (unknown)
@ 0x0 (unknown)
The full output of the train is here.
Your solver file has this line
lr_policy: "fixed"
But the Caffe output has this line
lr_policy: "step"
If you use stepwise learning, you must define stepsize. Again, your solver file indicates that you have defined stepsize, but Caffe output shows no stepsize. Please check your solver file again, and add this line back
stepsize: 10000