Trouble evaluating test data with Tensorflow's Object Detection API

Question

Overview

I'm working on an object detector using Tensorflow's Object Detection API using custom training data. My understanding so far has been that I can feed my training data into the model and then use a trained model[0] to evaluate test data, and that following that evaluation, I'd be able to see a set of images like the following to figure out what the trained model was able to detect in each image in my test data.

What I've done so far

Operating with this assumption, I've been able to create a training dataset in .tfrecord format, and I've been able to feed this into my model trainer using the following command:

PIPELINE_CONFIG_PATH="nbl-tf-test.config"
MODEL_DIR="./object_detection/modeldir"
NUM_TRAIN_STEPS=50000
SAMPLE_1_OF_N_EVAL_EXAMPLES=1

python3 object_detection/model_main.py \
    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
    --model_dir=${MODEL_DIR} \
    --num_train_steps=${NUM_TRAIN_STEPS} \
    --sample_1_of_n_eval_examples=$SAMPLE_1_OF_N_EVAL_EXAMPLES \
    --alsologtostderr

This gave me output that looks like this (repeated lots of times—this is just a representative sample):

2019-01-08 00:47:31.007154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2019-01-08 00:47:31.007931: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-01-08 00:47:31.007957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0
2019-01-08 00:47:31.007964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N
2019-01-08 00:47:31.008119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5278 MB memory) -> physical GPU (device: 0, na
me: Tesla K20Xm, pci bus id: 0000:02:00.0, compute capability: 3.5)
creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=17.90s).
Accumulating evaluation results...
DONE (t=2.83s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.007
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.020
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.003
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.006
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.008
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.018
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.089
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.485
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.015
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.367
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.573
creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=17.94s).
Accumulating evaluation results...
DONE (t=2.86s).

After this training session, I found a set of files in the object_detection/modeldir directory:

$ ls modeldir
checkpoint                              model.ckpt-308.meta
eval_0                                  model.ckpt-485.data-00000-of-00001
events.out.tfevents.1546897788.moose55  model.ckpt-485.index
export                                  model.ckpt-485.meta
graph.pbtxt                             model.ckpt-664.data-00000-of-00001
model.ckpt-1000.data-00000-of-00001     model.ckpt-664.index
model.ckpt-1000.index                   model.ckpt-664.meta
model.ckpt-1000.meta                    model.ckpt-844.data-00000-of-00001
model.ckpt-308.data-00000-of-00001      model.ckpt-844.index
model.ckpt-308.index                    model.ckpt-844.meta

My Question

What should I do with the contents of this directory to test my testing data? I assume I have to have a .tfrecord file for all the testing data (similar to the file I made before for training data). Based on the model's documentation, I've been trying to run the following command:

python3 object_detection/inference/infer_detections.py \
  --input_tfrecord_paths=output.tfrecord \
  --output_tfrecord_path=detections.tfrecord \
  --inference_graph=object_detection/modeldir-8steps/graph.pbtxt \
  --discard_image_pixels

but when I do this, I run into the following error (running with both Python 2 and Python 3):

INFO:tensorflow:Reading graph and building model...
Traceback (most recent call last):
  File "object_detection/inference/infer_detections.py", line 96, in <module>
    tf.app.run()
  File "/usr/lib/python3.4/site-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection/inference/infer_detections.py", line 74, in main
    image_tensor, FLAGS.inference_graph)
  File "/home/jlittle/honors/5-tf-nbl-clean-setup/object_detection/inference/detection_inference.py", line 71, in build_inference_graph
    graph_def.MergeFromString(graph_content)
TypeError: 'str' does not support the buffer interface

Does anyone know:

1. if the overview I gave above is correct? If so,
2. if this is the correct evaluation python script I should be running? If it is,
3. if there's anything wrong with the data I'm feeding the script, that would give me the error above? And finally,
4. if this is the best way of going about this project, or if I'm way off base and should consider something else?

Thanks.

---

[0]: Is this the correct vocabulary, using model like this in two different ways?

Answer 1

My understanding was wrong. The model_main.py program is self-contained; it performs both the training and evaluation step, and I didn't need to do anything with that directory afterward. All that is needed to see the output of the test is to attach a TensorBoard instance to the model directory.