Search code examples
androidtensorflowpython-3.6object-detection-api

TensorFlow object detection model works properly with stock model, but fails with error about an implemented operation not being implemented

Using the TPU training mode on Google Cloud, I trained an SSD MobileNet V1 FPN model to recognize two types of objects. The model trained without errors, and I was able to evaluate in TensorBoard. Following conversion to TensorFlow Lite and attempting to run the model in the demo application for object detection, the model fails with an error regarding a custom operation being unimplemented, despite TensorFlow's documentation stating that the operation is implemented in Lite builds.

I trained the model using a TPU on Google Cloud, and downloaded the model from the storage bucket.

Next, I exported the model using the latest revision of the object detection model (paths are generic on purpose):

python -m object_detection/export_tflite_ssd_graph \
--pipeline_config_path=$PATH_TO_CONFIG_FILE \
--trained_checkpoint=model.ckpt-$CHECKPOINT \
--output_directory=$OUTPUT_DIR \
--add_postprocessing_op=true

Next, I converted the model using the latest 1.12 tag from TensorFlow's Git repository (using Bazel 0.21 to avoid Bazel errors):

bazel run -c opt //tensorflow/contrib/lite/toco:toco \
--incompatible_package_name_is_a_function=false \
-- \
--input_file=$OUTPUT_DIR/tflite_graph.pb \
--output_file=$OUTPUT_DIR/detect.tflite \
--input_shapes=1,640,640,3 \
--input_arrays=normalized_input_image_tensor \
--output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' \
--inference_type=FLOAT \
--allow_custom_ops

Next, I edited the Bazel build closure to include my model which I copied over to the application directory, and edited the DetectionActivity Java file to reference my model:

# out of context
    assets = [
        #"//tensorflow/contrib/lite/examples/android/app/src/main/assets:labels_mobilenet_quant_v1_224.txt",
        #"@tflite_mobilenet//:mobilenet_quant_v1_224.tflite",
        #"@tflite_conv_actions_frozen//:conv_actions_frozen.tflite",
        #"//tensorflow/contrib/lite/examples/android/app/src/main/assets:conv_actions_labels.txt",
        #"@tflite_mobilenet_ssd//:mobilenet_ssd.tflite",
        "//tensorflow/contrib/lite/examples/android/app/src/main/assets:detect.tflite",
        #"//tensorflow/contrib/lite/examples/android/app/src/main/assets:box_priors.txt",
        "//tensorflow/contrib/lite/examples/android/app/src/main/assets:pascal_labels.txt",
    ],
# out of context

private static final int TF_OD_API_INPUT_SIZE = 640;
private static final boolean TF_OD_API_IS_QUANTIZED = false;
private static final String TF_OD_API_MODEL_FILE = "file:///android_asset/detect.tflite";
private static final String TF_OD_API_LABELS_FILE = "file:///android_asset/pascal_labels.txt";

Finally, I built and deployed the application using the following commands:

bazel build -c opt --config=android_arm64 --cxxopt='--std=c++11' "//tensorflow/contrib/lite/examples/android:tflite_demo"
adb install -r bazel-bin/tensorflow/contrib/lite/examples/android/tflite_demo.apk

All the code I modified was otherwise stock in the repository at https://github.com/tensorflow/models/tree/master/research/object_detection.

I expect the result to be a working application, demonstrable by building the application without making any modifications to the program (stock from the repository).

The actual result is the application crashes immediately upon launch with the below error message, captured with Logcat:

2019-02-09 16:38:28.229 32716-32716/? E/AndroidRuntime: FATAL EXCEPTION: main
    Process: org.tensorflow.lite.demo, PID: 32716
    java.lang.RuntimeException: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Didn't find custom op for name 'ResizeNearestNeighbor' with version 1
    Registration failed.

        at org.tensorflow.demo.TFLiteObjectDetectionAPIModel.create(TFLiteObjectDetectionAPIModel.java:124)
        at org.tensorflow.demo.DetectorActivity.onPreviewSizeChosen(DetectorActivity.java:110)
        at org.tensorflow.demo.CameraActivity$5.onPreviewSizeChosen(CameraActivity.java:362)
        at org.tensorflow.demo.CameraConnectionFragment.setUpCameraOutputs(CameraConnectionFragment.java:401)
        at org.tensorflow.demo.CameraConnectionFragment.openCamera(CameraConnectionFragment.java:408)
        at org.tensorflow.demo.CameraConnectionFragment.access$000(CameraConnectionFragment.java:64)
        at org.tensorflow.demo.CameraConnectionFragment$1.onSurfaceTextureAvailable(CameraConnectionFragment.java:95)
        at android.view.TextureView.getHardwareLayer(TextureView.java:390)
        at android.view.TextureView.draw(TextureView.java:339)
        at android.view.View.updateDisplayListIfDirty(View.java:18150)
        at android.view.View.draw(View.java:18928)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4240)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4026)
        at android.view.View.updateDisplayListIfDirty(View.java:18141)
        at android.view.View.draw(View.java:18928)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4240)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4026)
        at android.view.View.draw(View.java:19203)
        at android.view.View.updateDisplayListIfDirty(View.java:18150)
        at android.view.View.draw(View.java:18928)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4240)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4026)
        at android.view.View.updateDisplayListIfDirty(View.java:18141)
        at android.view.View.draw(View.java:18928)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4240)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4026)
        at android.view.View.updateDisplayListIfDirty(View.java:18141)
        at android.view.View.draw(View.java:18928)
        at android.view.ViewGroup.drawChild(ViewGroup.java:4240)
        at android.view.ViewGroup.dispatchDraw(ViewGroup.java:4026)
        at android.view.View.draw(View.java:19203)
        at com.android.internal.policy.DecorView.draw(DecorView.java:825)
        at android.view.View.updateDisplayListIfDirty(View.java:18150)
        at android.view.ThreadedRenderer.updateViewTreeDisplayList(ThreadedRenderer.java:669)
        at android.view.ThreadedRenderer.updateRootDisplayList(ThreadedRenderer.java:675)
        at android.view.ThreadedRenderer.draw(ThreadedRenderer.java:783)
        at android.view.ViewRootImpl.draw(ViewRootImpl.java:3098)
        at android.view.ViewRootImpl.performDraw(ViewRootImpl.java:2912)
        at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:2465)
        at android.view.ViewRootImpl.doTraversal(ViewRootImpl.java:1453)
        at android.view.ViewRootImpl$TraversalRunnable.run(ViewRootImpl.java:6958)
        at android.view.Choreographer$CallbackRecord.run(Choreographer.java:911)
        at android.view.Choreographer.doCallbacks(Choreographer.java:723)
        at android.view.Choreographer.doFrame(Choreographer.java:658)
        at android.view.Choreographer$FrameDisplayEventReceiver.run(Choreographer.java:897)
        at android.os.Handler.handleCallback(Handler.java:790)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:164)
        at android.app.ActivityThread.main(ActivityThread.java:6626)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:811)
     Caused by: java.lang.IllegalArgumentException: Internal error: Cannot create interpreter: Didn't find custom op for name 'ResizeNearestNeighbor' with version 1
    Registration failed.

        at org.tensorflow.lite.NativeInterpreterWrapper.createInterpreter(Native Method)
2019-02-09 16:38:28.229 32716-32716/? E/AndroidRuntime:     at org.tensorflow.lite.NativeInterpreterWrapper.<init>(NativeInterpreterWrapper.java:70)
        at org.tensorflow.lite.Interpreter.<init>(Interpreter.java:175)
        at org.tensorflow.lite.Interpreter.<init>(Interpreter.java:163)
        at org.tensorflow.demo.TFLiteObjectDetectionAPIModel.create(TFLiteObjectDetectionAPIModel.java:122)
            ... 51 more

Here are all of the resources I can provide to help debug this issue:

Solution

  • I found the solution to the issue. Using TensorFlow v1.13.0-rc1 appears to resolve the issue at this point.

    This is because the ResizeNearestNeighbor operation for TensorFlow Lite did not exist until v1.13, and I realize that my main mistake was looking at the documentation for v1.13 despite using v1.12.