Search code examples
scalaapache-sparkkryo

Spark Kryo Registrator

I'm using kryo as serializer in spark 2.1 I have set registration as required but I have facing a problem, I can't register this class:

org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[]

I have tried with classOf[Array[Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation").type]] and other combinations but I don't succeed in the registration of this class.

I have tried as well creating a java class like register and registering the class as org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableFileStatus$SerializableBlockLocation[].class;

Any clue is appreciated, Thanks.

Solution

  • I have solved my question, I've used:

    kryo.register(ClassTag(Class.forName("org.apache.spark.sql.execution.datasources.PartitioningAwareFileIndex$SerializableBlockLocation")).wrap.runtimeClass)

    It allows register for the class name at runtime wrapped in an array.