Spark is telling me that the features column is wrong

Question

What could cause this error. I'm a bit lost. Everything that i have found doesn't help me.

Stack trace:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Column features must be of type struct<type:tinyint,size:int,indices:array<int>,values:array<double>> but was actually struct<type:tinyint,size:int,indices:array<int>,values:array<double>>.
at scala.Predef$.require(Predef.scala:224)
at org.apache.spark.ml.util.SchemaUtils$.checkColumnType(SchemaUtils.scala:43)
at org.apache.spark.ml.PredictorParams$class.validateAndTransformSchema(Predictor.scala:51)
at org.apache.spark.ml.classification.Classifier.org$apache$spark$ml$classification$ClassifierParams$$super$validateAndTransformSchema(Classifier.scala:58)
at org.apache.spark.ml.classification.ClassifierParams$class.validateAndTransformSchema(Classifier.scala:42)
at org.apache.spark.ml.classification.ProbabilisticClassifier.org$apache$spark$ml$classification$ProbabilisticClassifierParams$$super$validateAndTransformSchema(ProbabilisticClassifier.scala:53)
at org.apache.spark.ml.classification.ProbabilisticClassifierParams$class.validateAndTransformSchema(ProbabilisticClassifier.scala:37)
at org.apache.spark.ml.classification.ProbabilisticClassifier.validateAndTransformSchema(ProbabilisticClassifier.scala:53)
at org.apache.spark.ml.Predictor.transformSchema(Predictor.scala:144)
at org.apache.spark.ml.PipelineStage.transformSchema(Pipeline.scala:74)
at org.apache.spark.ml.Predictor.fit(Predictor.scala:100)
at classifier.Clasafie.trainModel_MPC(Clasafie.java:46)
at classifier.Clasafie.MPC_Classifier(Clasafie.java:75)
at classifier.Clasafie.main(Clasafie.java:30)

Code part:

public static MultilayerPerceptronClassificationModel trainModel_MPC(SparkSession session,JavaRDD<LabeledPoint> data)
{

     int[] layers = {784,800};
     MultilayerPerceptronClassifier model = new MultilayerPerceptronClassifier().setLayers(layers)
             .setSeed((long) 42).setBlockSize(128).setMaxIter(1000);

     Dataset<Row> dataset = session.createDataFrame(data.rdd(), LabeledPoint.class);

     return model.fit(dataset);

}

Answer 1

I think the problem is in using the LabelPoint class from correct package.

Check the full package and use the on from ml package and not from mllib.

I think, you are using -

org.apache.spark.mllib.regression.LabeledPoint

please use (introduced in spark v2.0.0)-

org.apache.spark.ml.feature.LabeledPoint