I have trained a custom object detection model using 750 images using ImageAI on Google Colab Pro about a month ago using TensorFlowGPU 1.13 and have roughly 30min/epoch training time. Now, when I train using the same dataset but with TensorFlowGPU 2.4.3 (ImageAI doesnt support old TF anymore), I am getting very little GPU usage (0.1GB) and 6 hour per epoch training times. I have tried training the same model on my local machine and I am getting very slow training times as well.
I am using the following imports (based on ImageAI documentation):
!pip install tensorflow-gpu==2.4.0 keras==2.4.3 numpy==1.19.3 pillow==7.0.0 scipy==1.4.1 h5py==2.10.0 matplotlib==3.3.2 opencv-python keras-resnet==0.2.0
!pip install imageai --upgrade
I am pulling my training data from Google Drive.
Is there anything I could be missing that could speed up my object detection training times on either Google Colab or my local machine? The slow training times is slowing my research down.
If you want full GPU usage, from my experience, you must revert back to previous versions of ImageAI and it's compatible packages. Here is a list of compatible packages that I have installed that work as of now (January 2021) on my local machine and Google Colab:
This fixed any issue caused by the most recent patch of ImageAI. I now am back to full GPU usage. Until the issue is patched, I suggest using the old version.