Search code examples
javaintellij-ideaignite

ignite: how to save and re-load trained model


Following is the piece of code i used to train my model. After that how and where can i save my model and read it back other than FileExporter class? is it only in a file or can i store it in a cache and access back?

IgniteCache<Integer, double[]> cache = ignite.getOrCreateCache("MLData_IRIS");

// extracting   sepal length, sepal width, petal length, petal width
IgniteBiFunction<Integer, double[], Vector> featureExtractor = new RangeExtractor(1, 5);
IgniteBiFunction<Integer, double[], Double> labelExtractor = new PointExtractor(0);

System.out.println(">>> Create new training dataset splitter object.");
TrainTestSplit<Integer, double[]> split = new TrainTestDatasetSplitter<Integer, double[]>()
    .split(0.5, 0.5);

IgniteBiPredicate<Integer, double[]> testData = split.getTestFilter();
IgniteBiPredicate<Integer, double[]> trainData = split.getTrainFilter();

// Set up the trainer
KMeansTrainer trainer = new KMeansTrainer()
    .withDistance(new EuclideanDistance())  //other metrics are HammingDistance, ManhattanDistance
    .withAmountOfClusters(3) // number clusters want to create
    .withMaxIterations(100)
    .withEpsilon(1.0E-4D)
    .withSeed(1234L);

long t1 = System.currentTimeMillis();

KMeansModel mdl = trainer.fit(
    ignite,
    cache,
    trainData,
    featureExtractor,
    labelExtractor
);

long t2 = System.currentTimeMillis();
System.out.println("time taken to build the model : " + (t2 - t1) + " ms");

System.out.println(">>> --------------------------------------------");
System.out.println(">>> trained model: " + mdl.toString(true));

Solution

  • For now Ignite have only this mechanism - FileExporter.

    But, for version 2.8 we already implemented model storage.

    Sample for saving model:

    ModelStorage storage = new ModelStorageFactory().getModelStorage(ignite);
            storage.mkdirs("/");
            storage.putFile("/my_model", serializedMdl);
    
            ModelDescriptor desc = new ModelDescriptor(
                "MyModel",
                "My Cool Model",
                new ModelSignature("", "", ""),
                new ModelStorageModelReader("/my_model"),
                new IgniteModelParser<>()
            );
            ModelDescriptorStorage descStorage = new ModelDescriptorStorageFactory().getModelDescriptorStorage(ignite);
            descStorage.put("my_model", desc);
    

    Sample for loading model:

    Ignite ignite = Ignition.ignite();
    
            ModelDescriptorStorage descStorage = new ModelDescriptorStorageFactory().getModelDescriptorStorage(ignite);
            ModelDescriptor desc = descStorage.get(mdl);
    
            Model<byte[], byte[]> infMdl = new SingleModelBuilder().build(desc.getReader(), desc.getParser());
    
            Vector input = VectorUtils.of(x);
    
            try {
                return deserialize(infMdl.predict(serialize(input)));
            }
            catch (IOException | ClassNotFoundException e) {
                throw new RuntimeException(e);
            }
    

    Where x - is vector of doubles and mdl - is model name.

    NOTE: this API will be available with release 2.8. But, you could try it right now if you will build Ignite from master branch.