Search code examples
scalaapache-sparkvectorrow

row to vector in scala spark

I have a row that is obtained in the following way:

val row_name = df.collect()(i)

how can I convert this row to a type vector so that it can be passed to fromML() as follows ?

val vector_name=org.apache.spark.mllib.linalg.Vectors.fromML(row_name)

thanks in advance!

Solution

  • You can use vectorAssembler:

    import org.apache.spark.ml.linalg.Vector
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.ml.feature.VectorAssembler

val df = Seq((1,2),(3,4)).toDF("col1","col2")

val va = new VectorAssembler().setInputCols(Array("col1","col2")).setOutputCol("vector")

val row0 = va.transform(df).select("vector").collect()(0).getAs[Vector](0)

val vector0 = Vectors.fromML(row0)
// vector0: org.apache.spark.mllib.linalg.Vector = [1.0,2.0]