I am training Oxford dataset using tutorial with ResNet101 Faster R-CNN.
I am running the training on my local machine with 1 GPU not using Google Cloud.
My question is may I know what will be the max iteration?
My step is already over than 13,000,000 and did not stop yet.
The original faster r-cnn could define max iteration size here https://github.com/rbgirshick/py-faster-rcnn/blob/master/tools/train_faster_rcnn_alt_opt.py#L80
, but I am not sure about TensorFlow object detection API.
I did not change any parameter except for input_path and fine_tune_checkpoint (which I am using COCO pre-trained data with ResNet).
I thought that the max iteration will be in config file https://github.com/tensorflow/models/blob/master/object_detection/samples/configs/faster_rcnn_resnet101_pets.config#L100, but it seems it only define learning rate after certain step.
As per the docs By default, the training job will run indefinitely until the user kills it. So, run the training and evaluation jobs simultaneously and kill the processes(early stopping based on the validation accuracy saturation.
Note: from Jonathan comment, you can also add the number of steps explicitly num_steps