ML.NET in Unity

Question

I cannot figure out how to use ML.NET in Unity.

What I did: Converted my project to be compatible with framework 4.x. Converted api compatibility level to framework 4.x. Made assets/plugins/ml folder and droped in Microsoft.ML apis with corresponding xmls. Marked all ml.dlls platform settings to be only 86_64 compatible (this was redundant).

I can now: Call ML apis and create MlContext, TextLoader, and do the training of a model. When a model is trained I can also evaluate the trained model, but...

I cannot: When trying to get a prediction out of the model I get an error: github comment on issue from 28.12.18 (there is also a whole project attached there, you can see the code there) The same code works in visual studio solution.

 public float TestSinglePrediction(List<double> signal, MLContext mlContext, string modelPath)
{
    ITransformer loadedModel;
    using (var stream = new FileStream(modelPath, FileMode.Open, FileAccess.Read, FileShare.Read))
    {
        loadedModel = mlContext.Model.Load(stream);
    }
    var predictionFunction = loadedModel.MakePredictionFunction<AbstractSignal, PredictedRfd>(mlContext);
    var abstractSignal = new AbstractSignal()
    {
        Sig1 = (float)signal[0],
        Sig2 = (float)signal[1],
        Sig3 = (float)signal[2],
        Sig4 = (float)signal[3],
        Sig5 = (float)signal[4],
        Sig6 = (float)signal[5],
        Sig7 = (float)signal[6],
        Sig8 = (float)signal[7],
        Sig9 = (float)signal[8],
        Sig10 = (float)signal[9],
        Sig11 = (float)signal[10],
        Sig12 = (float)signal[11],
        Sig13 = (float)signal[12],
        Sig14 = (float)signal[13],
        Sig15 = (float)signal[14],
        Sig16 = (float)signal[15],
        Sig17 = (float)signal[16],
        Sig18 = (float)signal[17],
        Sig19 = (float)signal[18],
        Sig20 = (float)signal[19],
        RfdX = 0

    };
    var prediction = predictionFunction.Predict(abstractSignal);
    return prediction.RfdX;
}

This is the method that returns an error line: var predictionFunction = loadedModel.MakePredictionFunction<AbstractSignal, PredictedRfd>(mlContext);

Answer 1

As follows is a bit modyfied Iris Example from https://learn.microsoft.com/en-us/dotnet/machine-learning/tutorials/iris-clustering (that one does not work anymore due to some ML API changes)

1. First make sure that you have the latest .net version installed and that your Unity version is at least 2019.2.0f1 (this was a preview version) or higher.
2. Creste a new unity project. Create a Plugins folder inside your Assets folder. Import all ML .Net APIs into that folder (Might be a foolish thing to do, but I have forehand created a visual studio soution and added all those APIs to that solution via nuget, and than just copied those dll files to Assets/Plugins folder in my unity project).
3. In Assets folder create an Data folder and paste iris.data file from https://github.com/dotnet/machinelearning/blob/master/test/data/iris.data into it.

4. Create a script named MLuTest and paste into it the following code:

   public class MLuTest : MonoBehaviour{

       static readonly string _dataPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "iris.data");
       static readonly string _modelPath = Path.Combine(Environment.CurrentDirectory, "Assets", "Data", "IrisClusteringModel.zip");
       MLContext mlContext;
       void Start()
       {
           Debug.Log("starting...");
           mlContext = new MLContext(seed: 0);
           IDataView dataView = mlContext.Data.ReadFromTextFile<IrisData>(_dataPath, hasHeader: false, separatorChar: ',');
           string featuresColumnName = "Features";
           var pipeline = mlContext.Transforms
               .Concatenate(featuresColumnName, "SepalLength", "SepalWidth", "PetalLength", "PetalWidth")
               .Append(mlContext.Clustering.Trainers.KMeans(featuresColumnName, clustersCount: 3));//read and format flowery data
           var model = pipeline.Fit(dataView);//train
           using (var fileStream = new FileStream(_modelPath, FileMode.Create, FileAccess.Write, FileShare.Write))//save trained model
           {
               mlContext.Model.Save(model, fileStream);
           }
           var predictor = mlContext.Model.CreatePredictionEngine<IrisData, ClusterPrediction>(model);//predict
           IrisData Setosa = new IrisData
           {
               SepalLength = 5.1f,
               SepalWidth = 3.5f,
               PetalLength = 1.4f,
               PetalWidth = 0.2f
           };
           Debug.Log(predictor.Predict(Setosa).PredictedClusterId);
           Debug.Log("...done predicting, now do what u like with it");
       }
   }
   
   public class IrisData
   {
       [LoadColumn(0)]
       public float SepalLength;
   
       [LoadColumn(1)]
       public float SepalWidth;
   
       [LoadColumn(2)]
       public float PetalLength;
   
       [LoadColumn(3)]
       public float PetalWidth;
   }
   public class ClusterPrediction
   {
       [ColumnName("PredictedLabel")]
       public uint PredictedClusterId;
   
       [ColumnName("Score")]
       public float[] Distances;
   }
   

This should work right out of the box ... well it did for me. Where you could mess up is when getting api files, they could be different version from mine or just some .net framework compatible. So get the content of my Plugins folder (mind you all those apis may not be necesery, do the cherrypicking yourselve):https://github.com/dotnet/machinelearning/issues/1886 It used to be (in previous unity versions) that some player settings had to be changed but i did not have to do it. But anhoo here are mine:

I hope this helps, since Unity update 19.2 I have not had any problems mentioned in previous posts in this thread.