Running multiple cassandra queries and getting result in one RDD in Spark 2.3

Question

I have a Sequence of Strings which I want to use in the where clause of my Cassandra queries. So there would be one query for each String in the Sequence.

idSeq.foreach(id => {
  val rdd1 = sc.cassandraTable("keyspace", "columnfamily").
  where("id = ?", id).
  limit(100)
})

So I have put a loop over my Sequence and I am running the query for each id in the Sequence. I want to combine all the results in one RDD and perform a map and a save operation on the combined RDD. I have tried creating an empty RDD and doing union but the RDD remains empty even after the loop and nothing gets saved. What is the correct way of doing this?

Answer 1

sc.union(idSeq.map(id => {
    sc.cassandraTable("keyspace", "columnfamily").where("id = ?", id).limit(100)
}))