Search code examples
serializationapache-sparkkryo

Spark kryo serialization register Datatype[]

To force spark to use kryo serialization the following option can be set spark.kryo.registrationRequired.

How can I register org.apache.spark.sql.types.DataType[]?

Class is not registered: org.apache.spark.sql.types.DataType[]

which is throwing an exception. But trying to register like classOf[org.apache.spark.sql.types.DataType[]] will not compile

Solution

  • I was confused with java and scala array notation.

    classOf[Array[org.apache.spark.sql.types.DataType]]

    is the correct registration.

    Nonetheless, this is a spark internal class which should already be registered by spark.