Search code examples
pythonmachine-learningtensorflowobject-detection

tensorflow object detection Fine-tuning a model from an existing checkpoint

I'm trying to train a model from an existing checkpoint following the these instructions.

I have configured the Object Detection Training Pipeline using the faster_rcnn_resnet101_voc07.config configuration.

In checkpoint section I have set the directory where are located the checkpoint files of the pretrained model faster_rcnn_resnet101_coco.tar.gz

Acording the to this issue the fine_tune_checkpoint can be a path to a directory containing the three files: (.data-00000-of-00001, .index, .meta).

So I set the path to the directory "/home/docs/car_dataset/models/model/train"

gradient_clipping_by_norm: 10.0
  fine_tune_checkpoint: "/home/docs/car_dataset/models/model/train"
  from_detection_checkpoint: true
  num_steps: 800000
  data_augmentation_options {
    random_horizontal_flip {
    }
  }

However when I execute the script for training:

python object_detection/train.py     --logtostderr\
--pipeline_config_path=/home/docs/car_dataset/models/model/faster_rcnn_resnet101_voc07.config\
--train_dir=/home/docs/car_dataset/models/model/train\
--num_gpus=2

I got the error:

tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file /home/docs/car_dataset/models/model/train: Failed precondition: /home/docs/car_dataset/models/model/train: perhaps your file is in a different file format and you need to use a different restore operator?

I have also tried setting the path to every file in the directory

fine_tune_checkpoint: "/home/docs/car_dataset/models/model/train/model.ckpt.meta"

but I get the error:

tensorflow.python.framework.errors_impl.DataLossError: Unable to open table file /home/docs/car_dataset/models/model/train/model.ckpt.meta: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?

What is the correct way to define a pre trained model in the pipeline configuration having the three files: (.data-00000-of-00001, .index, .meta).

Tensorflow version: 1.2.1

Solution

  • What you have to do is to specify the whole path without the ".meta", ".index" and ".data-00000-of-00001" extensions. In your case, this looks to be: "/home/docs/car_dataset/models/model/train/model.ckpt" (which you'll notice is more specific than the directory).