Performance of gpu delegate, nnapi of tensorflow lite are almost the same on android mobile

Question

I use following codes to change the delegate on my phone(G3226)

try {     
      if(delegateNum == 1){
        GpuDelegate delegate = new GpuDelegate();
        Interpreter.Options options = (new Interpreter.Options()).addDelegate(delegate);
        d.tfLite = new Interpreter(loadModelFile(assetManager, modelFilename), options);
      }else if(delegateNum == 2){
        NnApiDelegate delegate = new NnApiDelegate();
        Interpreter.Options options = (new Interpreter.Options()).addDelegate(delegate);
        d.tfLite = new Interpreter(loadModelFile(assetManager, modelFilename), options);
      }else{
        d.tfLite = new Interpreter(loadModelFile(assetManager, modelFilename));
      }      

    } catch (Exception e) {
      throw new RuntimeException(e);
    }

But the performance are almost the same, not sure what happens.

• Phone G3226

• TFLite versions

  implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly' implementation 'org.tensorflow:tensorflow-lite-gpu:0.0.0-nightly' implementation 'org.tensorflow:tensorflow-lite-support:0.0.0-nightly'

• Model link, it is a quantized SSD model for object detection(accuracy is not that good)

Possible reasons I guess:

1. This model is too small, not complicated enough for gpu or nnapi to show off
2. gpu or nnapi on my phone are wek
3. my phone do not support gpu or nnapi, so they drop back to cpu

If it is 3, how could I check my phone support gpu or nnapi or not?Thanks

Answer 1

A few things:

1. The quantized SSD Model is probably out-of-date, please look at these ones for better accuracy. You will have to convert them with these instructions to get the .tflite versions though.

2. SSD models have a big post-processing step (NMS), which doesn't get accelerated. So the difference for SSD models is usually lesser than simpler ones like MobileNet used for classification.

3. NNAPI only works on Android 8.1 or later - is that true for your phone? Also, NNAPI may not accelerate on all architectures, so theres that case.

4. GPU delegate doesn't support quantized models (yet).