How to get the confidence of a prediction from a MLModel (Machine Learning Model) in Swift 5

even though my code is pretty simplistic in what it is trying to do, basically I want to have the classifier model return an accuracy/confidence value for how accurate it thinks it's prediction is, on actual Xcode application (not playground) please.

I'm using a made-up Dog breed Classifier which is Text Classifier as an example:

Code:

do {

  if try DogClassiferModel().prediction(text: "value").confidence >= 90 {

  print("We have a high enough accuracy that this is the name of a dog breed")

  }

} catch let error {

  print(error)
}

I know in Xcode's playground you can do this with test data:

Code:

  let data = try MLDataTable(contentsOf: URL(fileURLWithPath: "Path"))

  let (trainingData, testingData) = data.randomSplit(by: 0.8, seed: 5)

  let testClassifier = try MLTextClassifier(trainingData: trainingData, textColumn: "text", labelColumn: "recognized")

  // Getting the testing evaluation.
  let evaluationMetrics = testClassifier.evaluation(on: testingData)
  let evaluationAccuracy = (1.0 - evaluationMetrics.classificationError) * 100

  // We can print the accuracy with print(evaluationAccuracy).

Thought: Maybe CoreML doesn't work like the way I am trying to do it I don't know?

Solution

When you write,

try DogClassiferModel().prediction(text: "value")

what is returned is a DogClassiferModelOutput object. If your model's output is named confidence, you can get it by writing:

if let output = try DogClassiferModel().prediction(text: "value") {
  print(output.confidence)
}

However, classifier models in Core ML are usually treated in a special way. They can return the label of the highest scoring class, or a dictionary containing the probabilities for all the labels.

The best way to find out how this works for your model, is to look at the mlmodel file in Xcode, click the arrow to go the automatically generated source code file, and look for the "Output" class. This will have one or more properties that you can access (like shown in the example above).