Why model.predict() from tensorflowjs always return same prediction?

Question

I am making an app in the ionic 3 framework for recognizing handwritten characters and in order to do so, I am using tensorflowjs. The problem is that when everything is settled, the prediction always return the same value.

The model (converted from keras with tensorflowjs-converter) is stored locally in assets/models/model.json.

The images that I am trying to predict are extracted from a HTMLCanvasElement in which you can paint the character and then the function getCanvas() retrieves the ImageData from it:

getCanvas() {
    let photo = document.getElementById('MyCanvas') as HTMLCanvasElement;
    let ctx = photo.getContext('2d');
    const data: ImageData = ctx.createImageData(300,300);
    return data;
}

Then, the input tensor [1, 32, 32, 3] of the model is calculated from data and afterwards, I feed it to the model with evaluateModel(). In that function I load the model.json with tf.loadModel() and I try to predict the class of the image:

import * as tf from '@tensorflow/tfjs';

async evaluateModel() {
    imageData: ImageData = this.getCanvas();
    const modelURL: string = './../../assets/models/model.json';

    let img: tf.Tensor<tf.Rank.R3> = tf.fromPixels(imageData);
    img = tf.image.resizeBilinear(img, [32, 32]);
    img = img.expandDims(0);

    await tf.loadModel(modelURL).then(model => {
        const output: any = model.predict(img);
        const results: number = output.argMax(1).dataSync()[0];
        console.log(results);
    }

Everything works fine, no errors whatsoever, but when logging the output prediction onto the console it always returns the same value. Also, the array of predictions is very flat, pointing out that the model in python hits 99,01% accuracy on test:

console.log(Array.from(output.dataSync())); 

17  /* Result of the argMax function on output */
[0.011652837507426739, 0.03457817807793617, 0.029257778078317642, 0.008851004764437675, 0.01691449247300625, 0.026485547423362732, 0.04974032938480377, 0.016473202034831047, 0.021701140329241753, 0.020724112167954445, 0.03173287212848663, 0.024661116302013397, 0.007072054781019688, 0.022814681753516197, 0.011404283344745636, 0.015105938538908958, 0.024694452062249184, 0.07453715801239014, 0.011547397822141647, 0.03946337103843689, 0.018134022131562233, 0.027423541992902756, 0.014102200977504253, 0.016702469438314438, 0.05513478443026543, 0.030478181317448616, 0.012863627634942532, 0.011269242502748966, 0.022525735199451447, 0.022545555606484413, 0.02840271405875683, 0.011758353561162949, 0.006561313755810261, 0.020660076290369034, 0.009705542586743832, 0.024312887340784073, 0.011940978467464447, 0.020643217489123344, 0.009319263510406017, 0.00957920216023922, 0.01844164915382862, 0.015434195287525654, 0.02170679345726967, 0.017867043614387512, 0.013763428665697575, 0.029312126338481903]

The only two things that come to my mind when trying to solve this problem are:

1. The ImageData is not correctly extracting the canvas drawing and because of that the prediction is always returning the same value.
2. The weights on the model.json are not being accessed or loaded, therefore the model returns the same value.

Any ideas?

Any help is appreciated.

Answer 1

createImageData creates a new ImageDate with transparent black pixel. Therefore, each time that the method is called, the same ImageData will be returned.

You rather want to redraw the initial image on another canvas whose dimensions (height and width) matches the shape of the tensor given as parameter to the model.

// create a new canvas on which the initial canvas will be redrawn 
const canvas = document.createElement('canvas');
// draws on the first 32 * 32 pixels
canvas.drawImage(photos, 0, 0, 32, 32);
// returns the imageData corresponding to the part of the canvas we drew
return canvas.getImageData(0, 0, 32, 32);

With this new imageData, it is no longer necessary to use tf.bilinear, tf.reshape is enough to give to the tensor the required shape.

img = tf.image.reshape(img, [1, 32, 32, 3]);