Search code examples
scalatwitterstreamingapache-spark

How to run twitter popular tags of Spark streaming using scala?

I am new to Spark so please guide.

There are lots of example available related to Spark streaming using Scala.

You could check it out from https://github.com/apache/incubator-spark/tree/master/examples/src/main/scala/org/apache/spark/streaming/examples.

I want to run TwitterPopularTags.scala.

I am not able to set the twitter login details for this example.

http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html#linking-with-spark-streaming

I am successfully run the network count example.

But when i execute
./run-example org.apache.spark.streaming.examples.TwitterPopularTags local[2] then it will show me authentication failure issue...

I set twitter login details before initializing string context in TwitterPopularTags.scala like 

 System.setProperty("twitter4j.oauth.consumerKey", "####");
 System.setProperty("twitter4j.oauth.consumerSecret", "##");
 System.setProperty("twitter4j.oauth.accessToken", "##");
 System.setProperty("twitter4j.oauth.accessTokenSecret", "##");

Please guide.

Solution

  • Put the file "twitter4j.properties" into the Spark root directory (e.g. spark-0.8.0-incubating) before you run the Twitter examples.

    twitter4j.properties:

    oauth.consumerKey=***
oauth.consumerSecret=***
oauth.accessToken=***
oauth.accessTokenSecret=***

    Worked for me on Mac with the Scala examples.