How to do writeStream a dataframe in console? (Scala Spark Streaming)

Question

I want to debug my notebook thus I need to print out the streaming-data in notebook console mode. I have two questions: 1- Is it possible to do:

df.writeStream.format("console").start().awaitTermination()

2- If yes, where can I see the output?

This is after 10 minutes ... No error or result

Thanks! 🙏

Answer 1

I figured out what the problem in my case was. I had to add the ".outputMode("append")" in my method. Here is how it looks:

def writeStreamData(dataFrame: DataFrame): Unit = {
        /**
         * write the given dataframe into a file or console
         :params: dataframe
         */
        dataFrame.writeStream
            .format("console")
            .outputMode("append")
            .start()
            .awaitTermination()
    }