Search code examples
javascriptreactjstensorflowtensorflow.jsautoml

Tensorflow automl model in react

I'm trying to move a tensorflow model from its original html into a react app (built with create-react-app).

My App.js looks like this:

import logo from './logo.svg';
import * as tf from "@tensorflow/tfjs";
// import { loadImageclassification } from "@tensorflow/tfjs";
import './App.css';
import * as automl from "@tensorflow/tfjs-automl";
import * as modelJSON from './model.json';

function App() {

var loadFile = function(event) {
    var image = document.getElementById('output');
    image.src = URL.createObjectURL(event.target.files[0]);
  run();
};

async function run() {
  console.log(modelJSON);
        // const model = await tf.loadImageclassification('model.json');
        const model = await automl.loadImageClassification(modelJSON);
        const image = document.getElementById('output');
        const predictions = await model.classify(image);
        console.log(predictions);

        const pre = document.getElementById('result');
        pre.textContent = JSON.stringify(predictions, null, 2);
}

  return (
  <div className="App">
    <div className="hero-text">
      <h1>classifier</h1>
      <h3>Upload a picture to see what type it is! </h3>
      <p>
        <input type="file"  accept="image/*" name="image" id="file"  onChange={loadFile} />
      </p>
      <div id="demobox">
        <p>
          <label htmlFor="file">Upload your image</label>
        </p>
      </div> 
      <p><img id="output" width="200" alt="output" /></p>
      <div className="result" id="result">
      </div>
    </div>
  </div>
  );
}

export default App;

My index.html looks like this:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="theme-color" content="#000000" />
    <meta
      name="description"
      content="Web site created using create-react-app"
    />
    <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
    <link rel="manifest" href="%PUBLIC_URL%/manifest.json" />
  </head>
  <body>
    <noscript>You need to enable JavaScript to run this app.</noscript>
    <div id="root"></div>
  </body>
</html>

I am getting the following error, which seems to be issuing from somewhere in the loadImageClassification method:

Unhandled Rejection (TypeError): modelUrl.lastIndexOf is not a function

Edit:

Apparently loadImageClassification uses a fetch request under the hood and so requires a remote file (which is strange, because it seemed to work fine in the static index.html original version of this same project).

So I am now trying it just with a localhost express server, which at present looks like this:

const modelJSON = require('./model.json');

const express = require("express");
const bodyParser = require("body-parser");
const CORS = require("cors");

const app = express();

app.use(bodyParser.json());
app.use(CORS());

let modelObj = modelJSON;

app.get("/", (req, res) => {
  // console.log(modelObj);
  res.send(modelObj);
});

app.listen(5000, () => {
  console.log("Server listening on port 5000");
});

I can see the correct data when I navigate to localhost5000, but when I change

async function run() {
  const model = await automl.loadImageClassification(modelJSON);

to

async function run() {
  const modelUrl = "http://localhost:5000/";
  const model = await automl.loadImageClassification(modelUrl);

I get these errors:

enter image description here

Edit 2:

My server.js file now looks like this:

enter image description here

This produces the same errors as in the previous screenshot. (I am leaving in the comments that mess of an attempt to include all the shard files in this server.js file screenshot just because it may illustrate that I don't understand how to pass those ancillary model files to loadImageClassification when it makes its fetch request.)

So presumably the problem now has to do with the fact that loadImageClassification assumes that the ...shard__of6.bin and dict files are in the same directory as the model.json file.

So the question may (?) be: how to simulate the file structure that it (i.e., loadImageClassification) is expecting within a remote node server.

Fundamental confusion:

I'm don't understand why, when loadImageClassification is in the original static html, it does not seem to require a remote url from which to fetch model.json — but then when I put it in my react app, it suddenly gives me this error: "Fetch API cannot load file:///Users///client/src/model.json. URL scheme must be 'http' or 'https' for CORS request."

Solution

  • What's the location of the model on your local device?

    Try changing

    const modelUrl = "http://localhost:5000/"

    to

    const modelUrl = 'model/model.json'

    if the model is in build/model/, or to whatever the location is.