convert base64 image to tensor

Question

I'm sending the image as base64 string to the node express server to analysis the object detection with tensorflow. How to change the base64 image as tensors for object detection with cocossd model in node js.

Answer 1

server side NodeJs

The base64 string can be converted to binary and then be read as tensor using tf.node

 const b = Buffer.from(base64str, 'base64')
    // get the tensor
 const t = tf.node.decodeImage(b)

If other properties/values are not sent along the request, it would be best to send directly the image as binary in a post request or in a websocket. In that case, there would be no need to redo the conversion from base64 server side

browser side

const b = atob(base64str)
let byteNumbers = new Array(b.length);
for (let i = 0; i < b.length; i++) {
    byteNumbers[i] = b.charCodeAt(i);
}
let tensor = tf.tensor(byteNumbers)

This first option is synchronous. For big image it can possibly freeze the main thread. To alleviate that, this operation can be done in a web-worker.

The other option would be to create an image element and set it href attribute to the base64str and then use tf.browser.fromPixels

function load(url){
  return new Promise((resolve, reject) => {
    const im = new Image()
        im.crossOrigin = 'anonymous'
        im.src = 'url'
        im.onload = () => {
          resolve(im)
        }
   })
}

// use the load function inside an async function   

(async() => {
     const image = await load(url)
     let tensor = await tf.browser.fromPixels(image)
   })()