Does Vespa support binary classifier model serving at feed time?

Question

We want to classify documents as they are fed to Vespa using the API, and write those classification scores to document fields.

I'm not sure if it's possible to simply add the ONNX model into Vespa's application package directory, have that model classify any text fed to Vespa, and then write those classifications as document fields.

Does Vespa support model serving at feed time in this way?

Answer 1

Yes, you can do that with Vespa.

A custom document processor that reads the input, invokes the model, and stores the model's output in a new field. Stateless model evaluation

This example is a good starting point DimensionReductionDocProc, a document processor that uses the stateless model evaluation support to perform dimensionality reduction of a vector.

Then you need to export your classifier model to onnx format, and put it in the models folder in the application folder. If you wrap the inference in a component that can be shared between search and docproc and call it Classifier the services.xml of the container cluster looks something like this, plus the ClassifyDocProc.

<container id='default' version='1.0'>
    <nodes count='1'/>
    <component id='ai.vespa.examples.Classifier'/>
    <model-evaluation>
      <onnx>
        <models>
          <model name="classifier">
            <intraop-threads>1</intraop-threads>
          </model>
        </models>
      </onnx>
    </model-evaluation>
    <search/>
    <document-api/>
    <document-processing>
      <chain id='classifier' inherits='indexing'>
        <documentprocessor id='ai.vespa.examples.docproc.ClassifyDocProc'/>
        <documentprocessor 
      </chain>
    </document-processing>
  </container>