Search code examples
apache-sparkapache-spark-sqlkryo

spark kryo registration class missing

When forcing kryo registration via spark.kryo.registrationRequired", "true" in spark 2.2 my error is

Class is not registered: org.apache.spark.sql.catalyst.InternalRow[]

even when using:

.registerKryoClasses(
          Array(classOf[scala.collection.mutable.WrappedArray.ofRef[_]],
                classOf[org.apache.spark.sql.catalyst.InternalRow])
        )

Obviously InternalRow[] would not compile in scala. What am I missing here?

Solution

  • How to register InternalRow with Kryo in Spark is leading me to the correct answer.

    classOf[Array[org.apache.spark.sql.catalyst.InternalRow]]

    is required. However then, further spark classes need to be added as you move forward.