Search code examples
scalahadoopapache-sparkspark-streaming

spark streaming multiple sockets sources


I am pretty new to Spark. For my project I need to combine data coming from different streams on different ports. To test that I did an exercise which aim is to print data coming from streams from different ports. Below you can see the code:

object hello {
  def main(args: Array[String]) {

    val ssc = new StreamingContext(new SparkConf(), Seconds(2))
    val lines9 = ssc.socketTextStream("localhost", 9999)
    val lines8 = ssc.socketTextStream("localhost", 9998)

    lines9.print()
    lines8.print()

    ssc.start()
    ssc.awaitTermination()
  }
}

Than I run those code and start nc -lk 9999 and nc -lk 9998. When I put anything on port 9999 I see the output on Spark- works fine. When I put anything on 9998 I don't see any output.

Could you please explain me why there is no output on 9998 and how should I implement this to combine those two streams?


Solution

  • Ok, I find an answer for my problem. It is so simple that I feel silly to post it here.

    The problem was in executing application. I was doing it by: ./spark-submit --class hello while proper way to do this was: ./spark-submit --class hello --master local[2]

    Anyway, thanks guys for contribute.