I am Very new to Spark Machine Learning (4 days old) i am executing the below code in Spark Shell i am trying to predict some value
My requirement is I have data which consist of following
Columns
Userid,Date,SwipeIntime
1, 1-Jan-2017,9.30
1, 2-Jan-2017,9.35
1, 3-Jan-2017,9.45
1, 4-Jan-2017,9.26
2, 1-Jan-2017,9.37
2, 2-Jan-2017,9.35
2, 3-Jan-2017,9.45
2, 4-Jan-2017,9.46
I need to predict what will be the SwipeIntime for Userid = 1 will come on Date 5-Jan-2017 or any date
What i have tried is the below code in Spark Shell
Code:
case class LabeledDocument(Userid: Double, Date: String, label: Double)
val training = spark.read.option("inferSchema", true).csv("/root/Predictiondata2.csv").toDF
("Userid","Date","label").toDF().as[LabeledDocument]
import scala.beans.BeanInfo
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.ml.Pipeline
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.feature.{HashingTF, Tokenizer}
import org.apache.spark.mllib.linalg.Vector
import org.apache.spark.sql.{Row, SQLContext}
val tokenizer = new Tokenizer().setInputCol("Date").setOutputCol("words")
val hashingTF = new HashingTF().setNumFeatures(1000).setInputCol(tokenizer.getOutputCol).setOutputCol("features")
import org.apache.spark.ml.regression.LinearRegression
val lr = new LinearRegression().setMaxIter(10).setRegParam(0.3).setElasticNetParam(0.8)
val pipeline = new Pipeline().setStages(Array(tokenizer, hashingTF, lr))
val model = pipeline.fit(training.toDF())
case class Document(Userid: Integer, Date: String)
val test = sc.parallelize(Seq(Document(4, "04-Jan-18"),Document(5, "01-Jan-17"),Document(2, "03-Jan-17")))
model.transform(test.toDF()).show()
Getting Incorrect Output (Same SwipeIntime for all Users)
scala> model.transform(test.toDF()).show()
+------+---------+-----------+------------------+-----------------+
|Userid| Date| words| features| prediction|
+------+---------+-----------+------------------+-----------------+
| 4|04-Jan-18|[04-jan-18]|(1000,[455],[1.0])|9.726888888888887|
| 5|01-Jan-17|[01-jan-17]|(1000,[595],[1.0])|9.726888888888887|
| 2|03-Jan-17|[03-jan-17]|(1000,[987],[1.0])|9.726888888888887|
+------+---------+-----------+------------------+-----------------+
I would be thankful if anyone provide any suggestion to above code to make things working.
Why you think it's not working? Because the prediction are all the same?
I had a similar issue as describe here but in PySpark.
I solved it by raising the MaxIter and lowering RegParam and ElasticNetParam.
Try setting them this way:
val lr = new LinearRegression().setMaxIter(100).setRegParam(0.001).setElasticNetParam(0.0001)
Hope it works!