Search code examples
azure-stream-analytics

Clarification on how to calculate watermark for Azure Streaming Analytics

I'm working on understanding watermarks in Azure Streaming Analytics. Per MS documentation, there are two ways that watermarks can be calculated.

https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-time-handling#how-time-progresses-in-azure-stream-analytics

The second method states: "When there's no incoming event, the watermark is the current estimated arrival time minus the late arrival tolerance window. The estimated arrival time is the time that has elapsed from the last time an input event was seen plus that input event's arrival time."

Questions:

  • What is meant by "No incoming events"? Does this mean that the source (ex. Event Hub) is verified to be empty?

  • What is meant by "The last time an input event was seen."? Does this mean when it has exited the processing engine to the source?

Currently, this is how I interpret the calculation: enter image description here (7) Watermark = (5) [Estimated Arrival Time] - (6) [Late Arrival Tolerance Window]

(5) Estimated Arrival Time = (1) [Elapsed Time] + (4) [Last Arrival Time]

(1) Elapsed Time = Time that elapsed from the (2) [Last time an input event was last seen] and (3) [the current time]

Solution

  • Yes "No incoming events" means that ideally there is nothing to process on the Eventhub .

    For the second part , i think you have already gone through the doc which states

    "When there's no incoming event, the watermark is the current estimated arrival time minus the late arrival tolerance window. The estimated arrival time is the time that has elapsed from the last time an input event was seen plus that input event's arrival time." .

    If we are in the 0:45 mark and if you see that the expected events was at :40 unit ( lets assume that the event is coming in every 5 unit of time ) . so the watermark will be (45-15 (6) in your example)

    https://learn.microsoft.com/en-us/answers/questions/42145/clarification-on-how-to-calculate-watermark-for-az.html