Search code examples
scalaapache-sparkapache-kafka-streams

only showing top 20 rows

I have a streaming dataset. It reads from Kafka stream and writes into console.

 val outputStream = inputStream.writeStream.format("console")
                    .option("truncate", value = false)
                    .option("checkpointLocation", "checkpoint")

 outputStream.start()

 spark.streams.awaitAnyTermination()

However when there are more than one record, I get a message on console : only showing top 20 rows

And it shows only 20 records. Is there a way to retrieve all records ??

Spark version : 3.1.2

Solution

  • There is an option numRows, for example: 10000.

    val outputStream = inputStream.writeStream.format("console")
                    .option("truncate", value = false)
                    .option("numRows",10000)
                    .option("checkpointLocation", "checkpoint")