Search code examples
scalaapache-sparkdatabase-schema

Encoders.product[of a scala trait ].schema in spark

How to create a schema for spark from a trait? Considering a trait:

trait A{
val name:String
val size:String
}

As :

Encoders.product[A].schema

gives:

Error:type arguments do not conform to method product's type parameter bounds [T <: Product]

Also the number of fields will be more then the limit of case class parameters > 200

Solution

  • Case class do supports more than 22 columns, try creating outside all other class/object. If your need is to create a dataframe schema with large number of fields, this should work.

    val schema: StructType = StructType(
    Array(
      StructField(name = "name", StringType),
      StructField(name = "size", StringType)
    )
 )
val data = Seq(Row("Ramanan","29"))
spark.createDataFrame(spark.sparkContext.parallelize(data),schema).show()