Search code examples
scalaapache-sparkimplicit

Resolving implicit parameter at runtime

I know implicits are resolved during compilation time, so I guess I'm looking for a workaround for the following. Let's say I have a method with this signature:

class MyClass {
  def run[X](x: X)(implicit runnable: Runnable[X]) = runnable(x)
}

I want to wrap it:

class MyClassWrapper[Z: ClassTag] {
  val rdd: RDD[Z] = //Spark RDD
  def runWrapper[X](fun: Z => X) = rdd.foreach( (el:Z) => new MyClass.run(fun(el)))
}

This will not compile complaining about not being able to find an implicit. I could change the signature to:

def runWrapper[X](fun: Z => X)(implicit runnable: Runnable[X])

And all would compile but I'm using Spark which does not allow not Serializable objects to be captured inside operations from the outside (in my case the run would capture the implicit parameter from runWrapper) so I do need to get/create that implicit inside the run instead of closing it in from outside. Is that possible?

Solution

  • Couldn't come up with anything in runtime (didn't want to try reflection) so I ended up writing a few versions of the method, one per each type (since they don't share any common parent)

    class MyClassWrapper[Z: ClassTag] {
  val rdd: RDD[Z] = //Spark RDD
  def runWrapper(fun: Z => Implementation1) = rdd.foreach( (el:Z) => new MyClass.run(fun(el)))

  def runWrapper(fun: Z => Implementation2) = rdd.foreach( (el:Z) => new MyClass.run(fun(el)))
}