Search code examples
scalaapache-sparkkryo

How to register Receiver[] with Kryo?

In the process of porting a Spark 1.6 app to Spark 2.0.2, there's this complaint in the log:

com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.streaming.receiver.Receiver[]
Note: To register this class use: kryo.register(org.apache.spark.streaming.receiver.Receiver[].class);

This fails with Caused by: java.lang.ClassNotFoundException: org/apache/spark/streaming/receiver/Receiver[]/class:

sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver[].class")

This fails with Caused by: java.lang.ClassNotFoundException: org/apache/spark/streaming/receiver/Receiver[]:

sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver[]")

This fails with Class is not registered: org.apache.spark.streaming.receiver.Receiver[]:

sparkConf.set("spark.kryo.classesToRegister", "org.apache.spark.streaming.receiver.Receiver")

This fails with Class is not registered: org.apache.spark.streaming.receiver.Receiver[]:

sparkConf.registerKryoClasses(Array(
    classOf[org.apache.spark.streaming.receiver.Receiver[_]]
))

How can I get this class registered? I've been able to register other classes with Kryo, but not this one.

Edit:

In all these cases this setup is done:

sparkConf.set("spark.kryo.registrationRequired", "true")
sparkConf.set("spark.serializer", classOf[KryoSerializer].getName)
GraphXUtils.registerKryoClasses(sparkConf)
Solution

  • Found an approach that works here: Kryo serialization refuses to register class

    In short, change classOf[org.apache.spark.streaming.receiver.Receiver[_]], to classOf[Array[org.apache.spark.streaming.receiver.Receiver[_]]],