Using Spark 2.1.0 and spark-cassandra-connector 2.0.0-RC1-s_2.11.
Calling rdd.saveToCassandra("keyspace", "table", SomeColumns("A", "B"))
works fine when done inside the same function as the other transformations. However, defining a method on the same object as follows:
def saveToCassandra(rdd: RDD[_], keyspace: String, table: String, columns: SomeColumns) = {
rdd.saveToCassandra(keyspace, table, columns)
}
And calling it with saveToCassandra(rdd, "keyspace", "table", SomeColumns("A", "B"))
fails with:
Exception in thread "main" scala.ScalaReflectionException: <none> is not a term
at scala.reflect.api.Symbols$SymbolApi$class.asTerm(Symbols.scala:199)
at scala.reflect.internal.Symbols$SymbolContextApiImpl.asTerm(Symbols.scala:84)
at com.datastax.spark.connector.util.Reflect$.methodSymbol(Reflect.scala:12)
at com.datastax.spark.connector.util.ReflectionUtil$.constructorParams(ReflectionUtil.scala:63)
at com.datastax.spark.connector.mapper.DefaultColumnMapper.<init>(DefaultColumnMapper.scala:45)
at com.datastax.spark.connector.mapper.LowPriorityColumnMapper$class.defaultColumnMapper(ColumnMapper.scala:51)
at com.datastax.spark.connector.mapper.ColumnMapper$.defaultColumnMapper(ColumnMapper.scala:55)
Defining the helper methods with type tags for the RDD, e.g. rdd: RDD[(String, String)]
instead of RDD[_]
solve the problem. I guess that it has to do with reflection code inside the datastax connector and the way that it infers run-time types from the compile-time types. Apparently putting the underscore as any type erases type information for reflection. I'm kinda new to Scala and don't know how reflection works in Scala and if this behavior is desired or not.