Search code examples
scalaapache-sparkspark-graphx

Subtract an RDD from another RDD doesn't work correctly

I want to subtract an RDD from another RDD. I looked into the documentation and I found that subtract can do that. Actually, when I tested subtract, the final RDD remains the same and the values are not removed!

Is there any other function to do that? Or am I using subtract incorrectly?

Here is the code that I used:

 val vertexRDD: org.apache.spark.rdd.RDD[(VertexId, Array[Int])]
 val clusters  = vertexRDD.takeSample(false, 3)
 val clustersRDD: RDD[(VertexId, Array[Int])] = sc.parallelize(clusters)
 val final = vertexRDD.subtract(clustersRDD)
 final.collect().foreach(println(_))
Solution

  • Performing set operations like subtract with mutable types (Array in this example) is usually unsupported, or at least not recommended.

    Try using a immutable type instead.

    I believe WrappedArray is the relevant container for storing arrays in sets, but i'm not sure.