Search code examples
pythontensorflowdeep-learninggoogle-colaboratorymobilenet

My object detection model training process starts giving out mAP and AR as 0


I have been training an object detection model using ssd_mobilenet_v2_coco_2018_03_29.

Since last week, my colab .ipynb training script would give out mAP and AR of zero all the way till the end of training. [Picture of mAP and AR outputting zeros during training][1]

Until recently, I have successfully trained different versions of models without problems using the attached jupyter Training_Own_Data_Old.ipynb file below.

I checked the revision history to see if any changes were accidentally made, but there aren't any drastic changes except those mentioned below.

Some noticeable differences

Last week, I experienced the message- "Colab is experiencing issues connecting to Drive, and we are actively Investigating" whenever I try to connect to colab.

I also noticed that python-pil got updated. I installed the old version and tried again but that didn't solve the problem.

I've also checked the pipeline config files, training logs, tried with/without using dropout, just running the old .ipynb file itself without changes but still no luck.

I've attached github of the old working Training_Own_Data_Old.ipynb file an the new Training_Own_Data.ipynb file with errors. Logs are within each .ipynb file.

https://github.com/WalterYeYint/Jupyter-files

Could somebody please help? Any help is appreciated. [1]: https://i.sstatic.net/7oNT0.png


Solution

  • [SOLVED]: My script has been pulling tensorflow models repository everytime it runs. Downloaded that repository from the last time my script was working solved the problem.