What is "cudnn_status_internal_error" meaning Tensorflow pipeline?

Question

If we start an object_detection project using TensorFlow Model API, we got an error "cudnn_status_internal_error".

I'm using TensorFlow object detection pipeline to train the model.

How to solve it?

Answer 1

In my case, it is because the GPU resource is not enough.

To solve this issue in the pipeline, please add a piece of code in "model_main.py"

  session_config = tf.ConfigProto()
  #session_config.gpu_options.allow_growth = True
  session_config.gpu_options.per_process_gpu_memory_fraction = 0.8

  config = tf.estimator.RunConfig(model_dir=FLAGS.model_dir, session_config=session_config)