Search code examples
apache-flinkflink-streaming

Flink understanding late events vs watermark

Looking at this article it says regarding watermark

We will now set the watermark as current time - 5 seconds, which tells Flink to expect messages to be a maximum of 5 seconds dealy - This is because each window will be evaluated only when the watermark passes through it enter image description here

later in that post it explains that when setting allowed lateness :

Flink will not discard message unless it is past the window_end_time + allowed lateness

is actually causing a delayed evaluation of the window since allowed lateness is set ?

so what is the differnce actually in the usage of watermark and allowed lateness ? when to use which ?

Solution

  • The watermark delay sets a lower bound on how long Flink will wait for out-of-order events before triggering a window for the first time.

    The allowed lateness determines for how much longer Flink will keep around a window's state. Any late events that arrive while the window's state is still available will trigger the window again, causing it to produce updated results.

    Once the allowed lateness has expired, a window's state is purged, and any supremely late events are either dropped or sent to a side output.

    enter image description here

    If the downstream consumers of your window's output can deal with receiving updates for window results like this (e.g., the window is connected to a live dashboard), then it may make sense to set a relatively short watermark delay, and use allowed lateness liberally. On the other hand, if you can't take advantage of anything after the initial results, you'll want to make the watermark delay large enough to satisfy your requirements for accuracy/completeness, and set allowed lateness to zero.