I recerntly tried to build an Object Detection Android App using TFLite model. I built my own custom model (a Keras Model in HDF5 format) and converted the model succesfully into a custom TFLite model using the following command:
tflite_convert --keras_model_file=detect.h5 --output_file=detect.tflite --output_format=TFLITE --input_shapes=1,300,300,3 --input_arrays=normalized_input_image_tensor --output_arrays='TFLite_Detection_PostProcess','TFLite_Detection_PostProcess:1','TFLite_Detection_PostProcess:2','TFLite_Detection_PostProcess:3' --inference_type=QUANTIZED_UINT8 --mean_values=128 --std_dev_values=127 --change_concat_input_ranges=false --allow_custom_ops
I further added the associated MetaData to this particular model using this code:
import tensorflow as tf from tflite_support import metadata as _metadata
populator = _metadata.MetadataPopulator.with_model_file("detect.tflite") populator.load_associated_files(["labelmap.txt"]) populator.populate()
I then configured this model in the Android Package Example by tensorflow and made some tweaks to the Build.gradle file, DetectorActivity.java and TFLiteObjectDetectionAPIModel.java, respectively. I also made some UI changes according to what and how I needed it to look like. Additionally, I had to change the 'numBytesPerChannel' value for Float model from '4' to '3' since I was getting an error like this:
Cannot convert between a TensorFlowLite buffer with XYZ bytes and a ByteBuffer with ABC bytes
The build is successful yet the debugger throws me a fatal exception of "BufferOverFlowError".
11/13 14:57:02: Launching 'app' on Physical Device. Install successfully finished in 16 s 851 ms. $ adb shell am start -n "org.tensorflow.lite.examples.detection/org.tensorflow.lite.examples.detection.DetectorActivity" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER -D Waiting for application to come online: org.tensorflow.lite.examples.detection.test | org.tensorflow.lite.examples.detection Waiting for application to come online: org.tensorflow.lite.examples.detection.test | org.tensorflow.lite.examples.detection Connected to process 22667 on device 'samsung-sm_m315f-RZ8N50B0M5K'. Waiting for application to come online: org.tensorflow.lite.examples.detection.test | org.tensorflow.lite.examples.detection Connecting to org.tensorflow.lite.examples.detection Connected to the target VM, address: 'localhost:46069', transport: 'socket' Capturing and displaying logcat messages from application. This behavior can be disabled in the "Logcat output" section of the "Debugger" settings page. I/mples.detectio: Late-enabling -Xcheck:jni E/mples.detectio: Unknown bits set in runtime_flags: 0x8000 D/ActivityThread: setConscryptValidator setConscryptValidator - put W/ActivityThread: Application org.tensorflow.lite.examples.detection is waiting for the debugger on port 8100... I/System.out: Sending WAIT chunk I/System.out: Debugger has connected waiting for debugger to settle... I/chatty: uid=10379(org.tensorflow.lite.examples.detection) identical 1 line I/System.out: waiting for debugger to settle... I/System.out: waiting for debugger to settle... I/System.out: waiting for debugger to settle... I/System.out: waiting for debugger to settle... I/System.out: waiting for debugger to settle... I/System.out: waiting for debugger to settle... I/chatty: uid=10379(org.tensorflow.lite.examples.detection) identical 2 lines I/System.out: waiting for debugger to settle... I/System.out: waiting for debugger to settle... I/System.out: debugger has settled (1478) I/mples.detectio: Waiting for a blocking GC ClassLinker I/mples.detectio: WaitForGcToComplete blocked ClassLinker on ClassLinker for 7.502ms D/tensorflow: CameraActivity: onCreate org.tensorflow.lite.examples.detection.DetectorActivity@4d5b875 D/PhoneWindow: forceLight changed to true [] from com.android.internal.policy.PhoneWindow.updateForceLightNavigationBar:4274 com.android.internal.policy.DecorView.updateColorViews:1547 com.android.internal.policy.PhoneWindow.dispatchWindowAttributesChanged:3252 android.view.Window.setFlags:1153 com.android.internal.policy.PhoneWindow.generateLayout:2474 I/MultiWindowDecorSupport: [INFO] isPopOver = false I/MultiWindowDecorSupport: updateCaptionType >> DecorView@59812d[], isFloating: false, isApplication: true, hasWindowDecorCaption: false, hasWindowControllerCallback: true D/MultiWindowDecorSupport: setCaptionType = 0, DecorView = DecorView@59812d[] W/mples.detectio: Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed) W/mples.detectio: Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed) I/CameraManagerGlobal: Connecting to camera service D/VendorTagDescriptor: addVendorDescriptor: vendor tag id 3854507339 added I/CameraManagerGlobal: Camera 0 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client com.snapchat.android API Level 1 I/CameraManagerGlobal: Camera 1 facing CAMERA_FACING_FRONT state now CAMERA_STATE_CLOSED for client com.dolby.dolby234 API Level 2 I/CameraManagerGlobal: Camera 2 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client com.whatsapp API Level 1 I/CameraManagerGlobal: Camera 20 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client android.system API Level 2 I/CameraManagerGlobal: Camera 23 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client android.system API Level 2 I/CameraManagerGlobal: Camera 3 facing CAMERA_FACING_FRONT state now CAMERA_STATE_CLOSED for client com.sec.android.app.camera API Level 2 I/CameraManagerGlobal: Camera 4 facing CAMERA_FACING_FRONT state now CAMERA_STATE_CLOSED for client vendor.client.pid<4503> API Level 2 I/CameraManagerGlobal: Camera 50 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client com.sec.android.app.camera API Level 2 I/CameraManagerGlobal: Camera 52 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client android.system API Level 2 I/CameraManagerGlobal: Camera 54 facing CAMERA_FACING_BACK state now CAMERA_STATE_CLOSED for client android.system API Level 2 I/tensorflow: CameraActivity: Camera API lv2?: false D/tensorflow: CameraActivity: onStart org.tensorflow.lite.examples.detection.DetectorActivity@4d5b875 D/tensorflow: CameraActivity: onResume org.tensorflow.lite.examples.detection.DetectorActivity@4d5b875 I/ViewRootImpl@a101c3c[DetectorActivity]: setView = com.android.internal.policy.DecorView@59812d TM=true MM=false I/ViewRootImpl@a101c3c[DetectorActivity]: Relayout returned: old=(0,0,1080,2340) new=(0,0,1080,2340) req=(1080,2340)0 dur=31 res=0x7 s={true 532883185664} ch=true D/OpenGLRenderer: createReliableSurface : 0x7c1211ecc0(0x7c12502000) D/OpenGLRenderer: makeCurrent EglSurface : 0x0 -> 0x0 I/mali_winsys: new_window_surface() [1080x2340] return: 0x3000 D/OpenGLRenderer: eglCreateWindowSurface : 0x7c120c3600 I/CameraManagerGlobal: Camera 0 facing CAMERA_FACING_BACK state now CAMERA_STATE_OPEN for client org.tensorflow.lite.examples.detection API Level 1 I/tensorflow: CameraConnectionFragment: Desired size: 640x480, min size: 480x480 I/tensorflow: CameraConnectionFragment: Valid preview sizes: [1920x1080, 1440x1080, 1280x720, 1088x1088, 1024x768, 960x720, 720x720, 720x480, 640x480] I/tensorflow: CameraConnectionFragment: Rejected preview sizes: [800x450, 640x360, 352x288, 320x240, 256x144, 176x144] CameraConnectionFragment: Exact size match found. W/Gralloc3: mapper 3.x is not supported I/gralloc: Arm Module v1.0 W/Gralloc3: allocator 3.x is not supported D/OpenGLRenderer: makeCurrent EglSurface : 0x0 -> 0x7c120c3600 I/Choreographer: Skipped 34 frames! The application may be doing too much work on its main thread. I/ViewRootImpl@a101c3c[DetectorActivity]: MSG_WINDOW_FOCUS_CHANGED 1 1 D/InputMethodManager: prepareNavigationBarInfo() DecorView@59812d[DetectorActivity] D/InputMethodManager: getNavigationBarColor() -855310 D/InputMethodManager: prepareNavigationBarInfo() DecorView@59812d[DetectorActivity] D/InputMethodManager: getNavigationBarColor() -855310 V/InputMethodManager: Starting input: tba=org.tensorflow.lite.examples.detection ic=null mNaviBarColor -855310 mIsGetNaviBarColorSuccess true , NavVisible : true , NavTrans : false D/InputMethodManager: startInputInner - Id : 0 I/InputMethodManager: startInputInner - mService.startInputOrWindowGainedFocus I/ViewRootImpl@a101c3c[DetectorActivity]: MSG_RESIZED: frame=(0,0,1080,2340) ci=(0,83,0,39) vi=(0,83,0,39) or=1 D/InputMethodManager: prepareNavigationBarInfo() DecorView@59812d[DetectorActivity] getNavigationBarColor() -855310 V/InputMethodManager: Starting input: tba=org.tensorflow.lite.examples.detection ic=null mNaviBarColor -855310 mIsGetNaviBarColorSuccess true , NavVisible : true , NavTrans : false D/InputMethodManager: startInputInner - Id : 0 I/CameraManagerGlobal: Camera 0 facing CAMERA_FACING_BACK state now CAMERA_STATE_ACTIVE for client org.tensorflow.lite.examples.detection API Level 1 W/TFLiteObjectDetectionAPIModelWithInterpreter: cow1 cow2 cow3 cow4 W/TFLiteObjectDetectionAPIModelWithInterpreter: cow5 cow6 I/tflite: Initialized TensorFlow Lite runtime. I/tensorflow: DetectorActivity: Camera orientation relative to screen canvas: 90 I/tensorflow: DetectorActivity: Initializing at size 640x480 I/tensorflow: DetectorActivity: Preparing image 1 for detection in bg thread. W/System: A resource failed to call close. I/tensorflow: DetectorActivity: Running detection on image 1
E/AndroidRuntime: FATAL EXCEPTION: inference
Process: org.tensorflow.lite.examples.detection, PID: 22667
java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:542)
at java.nio.DirectByteBuffer.putFloat(DirectByteBuffer.java:809)
at org.tensorflow.lite.examples.detection.tflite.TFLiteObjectDetectionAPIModel.recognizeImage(TFLiteObjectDetectionAPIModel.java:187)
at org.tensorflow.lite.examples.detection.DetectorActivity$2.run(DetectorActivity.java:183)
at android.os.Handler.handleCallback(Handler.java:883)
at android.os.Handler.dispatchMessage(Handler.java:100)
at android.os.Looper.loop(Looper.java:237)
at android.os.HandlerThread.run(HandlerThread.java:67)
I/Process: Sending signal. PID: 22667 SIG: 9
Disconnected from the target VM, address: 'localhost:46069', transport: 'socket'
The error suggests a change in these lines:
In TFLiteObjectDetectionAPIModel.java:
private static final float IMAGE_MEAN = 127.5f;
private static final float IMAGE_STD = 127.5f;
//...
@override
protected void addPixelValue(int pixelValue) {
imgData.putFloat((((pixelValue >> 16) & 0xFF) - IMAGE_MEAN) / IMAGE_STD);
imgData.putFloat((((pixelValue >> 8) & 0xFF) - IMAGE_MEAN) / IMAGE_STD);
imgData.putFloat(((pixelValue & 0xFF) - IMAGE_MEAN) / IMAGE_STD);
}
In DetectorActivity.java:
@override
public void run() {
LOGGER.i("Running detection on image " + currTimestamp);
final long startTime = SystemClock.uptimeMillis();
final List<Detector.Recognition> results = detector.recognizeImage(croppedBitmap);
lastProcessingTimeMs = SystemClock.uptimeMillis() - startTime;
Please let me know if I missed any step or did anything wrong.
P. S. - I used a dull-trained model before this and the app worked just fine, except for the fact that it showed all the boundary boxes at once with negligible changes in any detections. I am currently using a well trained model which looks like this (via netron):
I have encountered this error before as well. Usually this happens when 2 things are not correct in your code that feeds data into your model:
The image size is incorrect (i.e. you are feeding a buffer larger than the one allocated as input buffer in your model)
The data type you are feeding the data as is not correct. (i.e. the model expects uint8 but you are feeding a buffer filled with float values).
If you converted the model via the tflite_convert tool, sometimes the inputs can change types between float or int (depending on your arguments)
Is there a way for you to check if either one of these cases are happening?
