Search code examples
pythonscalaapache-sparkrdd

Which function in spark is used to combine two RDDs by keys

Let us say I have the following two RDDs, with the following key-pair values.

rdd1 = [ (key1, [value1, value2]), (key2, [value3, value4]) ]

and 

rdd2 = [ (key1, [value5, value6]), (key2, [value7]) ]

Now, I want to join them by key values, so for example I want to return the following

ret = [ (key1, [value1, value2, value5, value6]), (key2, [value3, value4, value7]) ]

How I can I do this, in spark using Python or Scala? One way is to use join, but join would create a tuple inside the tuple. But I want to only have one tuple per key value pair.

Solution

  • I would union the two RDDs and to a reduceByKey to merge the values. 

    (rdd1 union rdd2).reduceByKey(_ ++ _)