I have an error while trying to test CNTK.
I'm trying to slice a parameter using input_variable as an index. Using C.gather for slicing will cause a memory error in the backprop process.
Errors occur in all cntk2 environments such as CPU, GPU, Docker, Local installation.
Error Msg and Callstack
RuntimeError: CUBLAS failure 11: CUBLAS_STATUS_MAPPING_ERROR ; GPU=0 ; hostname=.... ; expr=cublasGetMatrix((int) numRows, (int) numCols, sizeof(ElemType), Data(), (int) GetNumRows(), dst, (int) colStride)
[CALL STACK] Microsoft::MSR::CNTK::CudaTimer:: Stop - Microsoft::MSR::CNTK::Matrix:: CopySection - Microsoft::MSR::CNTK::Matrix:: AssignValuesOf - CNTK::NDArrayView:: CopyFrom - CNTK::NDArrayView::NDArrayView
- CNTK::TrainingParameterSchedule:: Serialize - CNTK::DictionaryValue:: Save - CNTK::Trainer:: SummarizeTrainingProgress - PyInit__cntk_py - PyCFunction_Call - PyEval_GetFuncDesc - PyEval_EvalFrameEx - PyEval_GetFuncDesc (x2) - PyEval_EvalFrameEx (x2)
Code
x = input_val[:-2]
p1 = input_val[-2]
p2 = input_val[-1]
activator = relu
W1 = C.Parameter((slices,input_dim,hidden_layers_dim), init=C.glorot_normal(), name='W1')
b1 = C.Parameter((slices,hidden_layers_dim), init=0, name='b1')
W2 = C.Parameter((slices,hidden_layers_dim,hidden_layers_dim), init=C.glorot_normal(), name='W2')
b2 = C.Parameter((slices,hidden_layers_dim), init=0, name='b2')
W3 = C.Parameter((slices,hidden_layers_dim,output_dim), init=C.glorot_normal(), name='W3')
b3 = C.Parameter((slices,output_dim), init=0, name='b3')
W11 = C.gather(W1, p1)
b11 = C.gather(b1, p1)
W1x = C.reshape(W11, (input_dim,hidden_layers_dim))
b1x = C.reshape(b11, (hidden_layers_dim,))
W21 = C.gather(W2, p1)
b21 = C.gather(b2, p1)
W2x = C.reshape(W21, (hidden_layers_dim,hidden_layers_dim))
b2x = C.reshape(b21, (hidden_layers_dim,))
W31 = C.gather(W3, p1)
b31 = C.gather(b3, p1)
W3x = C.reshape(W31, (hidden_layers_dim,output_dim))
b3x = C.reshape(b31, (output_dim,))
x = activator(C.times(x, W1x) + b1x)
x = activator(C.times(x, W2x) + b2x)
x = C.times(x, W3x) + b3x
I can't reproduce this with the latest master. This is very likely a bug that was fixed right after the CNTK 2.1 release. The next release (2.2) will be around Sept. 15 2017. If the problem persists, after you have upgraded to 2.2, opening a github issue might be the right way to address this.