Search code examples
scalaapache-sparkkryo

Default set of kryo registrations for spark/scala

I am running into a whack-a-mole with many classes requiring kryo registration. Is there a default registration for common spark classes that can help?

Here is a list of classes that I have had to add so far - and there is no end in sight:

  conf.registerKryoClasses(Array(classOf[Row]))
  conf.registerKryoClasses(Array(classOf[InternalRow]))
  conf.registerKryoClasses(Array(classOf[Array[InternalRow]]))
  conf.registerKryoClasses(Array(classOf[scala.reflect.ClassTag$$anon$1]))
  conf.registerKryoClasses(Array(classOf[org.apache.spark.sql.catalyst.expressions.UnsafeRow]))
  conf.registerKryoClasses(Array(classOf[Array[org.apache.spark.sql.types.StructType]]))
  conf.registerKryoClasses(Array(classOf[org.apache.spark.sql.types.StructType]))
Solution

  • This is not really an answer but is a partial explanation of the behavior. There was some older code that was forcing kryo to be particular:

     val conf: SparkConf = new SparkConf()
  .set("spark.kryo.registrationRequired", "true")

    I removed that line and then the "registration missing" complaints magically disappeared.