Search code examples
tensorflowobject-detection

tensorflow object detection API: training is very slow

I am currently studying google tensorflow object detection API. When I try to retrain the model with Oxford III pet dataset, the training process is very slow.

Here is what I found so far:

  1. most of time only 2% GPU is utilzed.
  2. but CPU utilization is 60%, so It seems GPU is not starved by input, otherwise CPU should be near 100% utilization.

I am trying to profile it with tensorflow profiler, but I am in a bit hurry now, any idea or suggestion would be helpful.

Solution

  • I found the problems. It's the issue with input, my tfrecord file is corrupted somehow, so the input thread hang up sometimes.