No implicits arguments of type T, Trying to cast a dataframe to a Dataset[T]

Question

I'm trying to build a generic function to consolidate some mongoDb collections, I'm using case classes to type the collections and on my function I receive the type as a parameter [T] like this:

  def refreshCollection[T](newDS:Dataset[T],oldDS:Dataset[T]): Dataset[T]={
    val filteredOldDS=oldDS.join(newDS, Seq("id"),"left_anti").as[T]
    filteredOldDS.union(newDS)
  }

The problem is when I try to convert the Dataframe result of the join to the original case class using .as[T] to return a Dataset[T] I've got this error even tough I've imported the sparkSession.implicits._:

no implicit arguments of type: Encoder[T]

The interesting part is, when I make the conversion with a fixed case class works fine, any advice on this?

Thanks in advance!

Answer 1

I believe it's because your code doesn't guarantee that there will be Encoder[T] available when necessary.

You can try to parametrise your with implicit decoder and postpone the moment when compiler will try to find the required decoder.

  def refreshCollection[T](newDS:Dataset[T],oldDS:Dataset[T])(implicit enc: Encoder[T]): Dataset[T]={
    val filteredOldDS=oldDS.join(newDS, Seq("id"),"left_anti").as[T]
    filteredOldDS.union(newDS)
  }

Of course, you will need to bring into scope somehow the Encoder[MyCaseClass] when calling refreshCollection