Sample word count application using Flume + Spark Streaming

Question

Below is the code I use to get Flume events and process in spark.streaming using Scala.

When trying to use reduceBykey function I get the following compilation error:

value reduceByKey is not a member of org.apache.spark.streaming.dstream.DStream[(String, Int)]

Why?

Do we need to handle Flume streams in any specific way other than this?

I don't think it's a dependency issue, I have other simple applications working in the same Eclipse IDE where reduceBykey is being used.

package com.deloitte.spark.learning

import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import org.apache.spark.streaming.flume._

object Wordcount {
    def main(args: Array[String]) {
        if (args.length < 2) {
            System.err.println("Usage: NetworkWordCount <hostname> <port>")
            System.exit(1)
        }
        val sparkConf = new Sparkconf().setMaster("local[2]").setAppName("aa")
        val ssc = new StreamingContext(sparkConf, Seconds(200))
        val stream = FlumeUtils.createStream(ssc, args(0), args(1).toInt)
        stream.count().map(cnt => "Received " + cnt + " flume events." ).print()
        val lines = stream.map {
            e => new String(e.event.getBody().array(), "UTF-8")
        }
        val words = lines.flatMap(_.split(" "))
        val wordCounts = words.map(x => (x, 1))
        ssc.start()
        ssc.awaitTermination(1000)
    }
}

Answer 1

In order to obtain the function reduceByKey on a DStream[(String, Int)] you need to import the following package:

import org.apache.spark.streaming.StreamingContext._