Search code examples
scalaapache-sparkspark-streaming

Dstream Spark Streaming


Hello I wrote the following code

val receivedStream: DStream[PubsubSparkMessage] =
      PubsubSource.createStream(ssc, pubsubConfig, pubsubSubscriber)
    @volatile var receivedMessages: List[PubsubSparkMessage] = List() //scalastyle:ignore
    receivedStream.foreachRDD { rdd =>
      if (!rdd.isEmpty()) {
        receivedMessages = List(rdd.collect)
        receivedMessages
      }
    }

I am getting an error because type mismatch; My question is how to modify List[Array[PubsubSparkMessage]] to List[PubsubSparkMessage]


Solution

  • Something like:

    val list : List[Array[PubsubSparkMessage]] = List()
    
    val result = list.flatMap(x => x)