Search code examples
scalaapache-sparkshapeless

Shapeless ToTraversable is not serialisable

I am trying to use HList from Shapeless in a Spark RDD, I would like to use the toList method which requires the following implicit:

implicit ev2: ToTraversable.Aux[InType, List, OutType]

However, if I pass this implicit to the RDD, it throws a NotSerializableException:

Caused by: java.io.NotSerializableException: scala.collection.generic.GenTraversableFactory$$anon$1
    at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
    at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
    at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
    at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)

Is there an alternative way to achieve toList in Shapeless, or a way to provide an implicit that is serialisable?

Solution

  • After reading throught some of the shapeless code I found in the tests that the workaround is to put this implicit CanBuildFrom in scope for ToTraversable:

      implicit def listSerializableCanBuildFrom[T]: CanBuildFrom[List[T], T, List[T]] =
    new CanBuildFrom[List[T], T, List[T]] with Serializable {
      def apply(from: List[T]) = from.genericBuilder[T]

      def apply() = List.newBuilder[T]
    }