Stop streaming context in Spark Streaming after a period of time

Question

I building an application which receives DStreams from Twitter, the only way to stop the Streaming context is by stoping the execution. I wonder if there is a way to set a time and terminate the streaming socket without stoping the entire application?

Answer 1

You can use either

awaitTerminationOrTimeout(long)

as mentioned in the previous answer, or you can stop the streaming context manually from your other thread:

    // in the main thread
    awaitTermination(); // will wait forever or until the context is stopped

    // in another thread
    streamingContext.stop();