Search code examples
pythontensorflowdeep-learninggoogle-colaboratorytraining-data

Assertion Error while training on last epoch

I have used a model from this repo. I want to train it again to improve performance. This is my training command.

!python train.py --snapshot efficientdet-d0.h5 --phi 0 --gpu 0 --weighted-bifpn --random-transform --compute-val-loss --freeze-bn --batch-size 4 --steps 100 coco datasets/coco

And the directory of folders:

enter image description here

The first 99 epochs work well. 

99/100 [============================>.] - ETA: 0s - loss: 0.3978 - classification_loss: 0.2983 - regression_loss: 0.0995datasets/coco/images/train2017/i.rf.4eba631a9cf8fef57df45f64cf16f258.jpg
datasets/coco/images/train2017/i.rf.3bb1c054edd9e0b91f8944b797c28eac.jpg
datasets/coco/images/train2017/i.rf.8dba641d2187f073238f34fddd5499fc.jpg
datasets/coco/images/train2017/i.rf.bfd984b934688cdca720056e6733219d.jpg
datasets/coco/images/train2017/i.rf.2d0be24e1ac8afdd69ebb9cf03367be5.jpg
datasets/coco/images/train2017/i.rf.a6a1177aa50ae605fe184b766421d4c4.jpg
100/100 [==============================] - ETA: 0s - loss: 0.3959 - classification_loss: 0.2966 - regression_loss: 0.0993datasets/coco/images/val2017/000000061960.jpg

Afterwards, I see this error:

Traceback (most recent call last):
  File "train.py", line 381, in <module>
    main()
  File "train.py", line 376, in main
    validation_data=validation_generator
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1482, in fit_generator
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 875, in fit
    return_dict=True)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py", line 1060, in evaluate
    model=self)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/data_adapter.py", line 1112, in __init__
    model=model)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/data_adapter.py", line 908, in __init__
    **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/data_adapter.py", line 772, in __init__
    peek, x = self._peek_and_restore(x)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/data_adapter.py", line 912, in _peek_and_restore
    return x[0], x
  File "/content/drive/My Drive/EfficientDet-master/generators/common.py", line 424, in __getitem__
    inputs, targets = self.compute_inputs_targets(group)
  File "/content/drive/My Drive/EfficientDet-master/generators/common.py", line 373, in compute_inputs_targets
    image_group = self.load_image_group(group)
  File "/content/drive/My Drive/EfficientDet-master/generators/common.py", line 225, in load_image_group
    return [self.load_image(image_index) for image_index in group]
  File "/content/drive/My Drive/EfficientDet-master/generators/common.py", line 225, in <listcomp>
    return [self.load_image(image_index) for image_index in group]
  File "/content/drive/My Drive/EfficientDet-master/generators/coco.py", line 132, in load_image
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.1.2) /io/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'
Solution

  • The error means that the image (from a file path) that is currently trying to be processed could no be found. If you look at the training output, you'll see that all the image paths come from datasets/coco/images/train2017/, but the very last line says is looking for a file in datasets/coco/images/val2017.

    The script is done training and is looking for validatioon/testing data, which I suspect it can't find. Check that the val2017 directory exists, and that there are images in the directory that can be loaded with cv.imread().