Search code examples
pythonamazon-web-servicesgpumxnet

Python Stopped Working on switiching to GPU context


I am trying to setup my machine learning environment on AWS as follows :-

OS: windows server 2012 r2 , 64 bit
instance: p2.xlarge
GPU : Tesla K80 series
CUDA: 9.2.148
Graphis driver: 398.26 (installed by cuda toolkit)
python : 3.5 ( tested using 2.7.15 as well) ,64 bit
IDE: Pycharm Community 2018.2, 64 bit
mxnet librabry: mxnet-cu92

now when I run following code :

import mxnet as mx
from mxnet import nd
mx.random.seed(1)
z = nd.ones(shape=(3,3), ctx=mx.cpu())
print(z)

it works fine , but when I change ctx=mx.gpu() or ctx=mx.gpu(0) , I get error python stopped working.

cuda setup is working fine I compiled and ran deviceQuery , bandwidthTest sample application, these are giving output as expected.

EDIT:: python crash details gives me following information :

Fault Module Name:  ucrtbase.DLL

I tried with Cuda Graphis driver: 398.44 (recommended from cuda website for my gpu and os ) still no luck so far.

Any idea for resolving this issue?


Solution

  • Finally I got the solution after spending nearly 3 days, it was mxnet library which was causing problem. After trying different cuda toolkit and mxnet libraries following combination worked for me :-

    1) Cuda Toolkit - 9.2
       Mxnet lbrary : mxnet-cu92 version 1.2.0b20180525
       (Failing for most of the recent versions ) 
    2)  Cuda Toolkit - 8.0
       Mxnet lbrary : mxnet-cu80 version 0.11.0
       (Failing for most of the recent versions  ) 
    

    thanks