Search code examples
sqlscalaapache-sparkbigdataspark-streaming

spark structured streaming sql window in eventTime


How would be the equivalent String SQL query for this very same query:

val streamingWindowDF = 
  streamingInputDF
    .groupBy(
      $"sessiondId",
      window($"time", "10 seconds","2 seconds"))

I would like to do the same but with SQL string something like:

val result = spark.sql(".....")

What I want to do is a sliding window.

Thanks.


Solution

  • window is a valid SQL function and can be used as such, i.e.

    SELECT COUNT(*)
    FROM df
    GROUP BY  window(time, '10 seconds', '2 seconds')
    

    or in plain SELECT

    SELECT *, window(time, '10 seconds', '2 seconds') AS w
    FROM df