I have a flink job (application mode) with a processing time window that aggregates events from multiple kafka topics. This works as expected for the happy path.
The happy path is a simple reduction of the events that fall into the window (before window expiry). A custom trigger is used to check the condition that required events, for the reduction, have show up. In this case a FIRE_AND_PURGE is returned, which cause the reduction to be published to a sink.
For a timer expiry scenario (and all the required events for that window haven't shown up yet), can the flink job generate an alternate output to the sink? Ideally, this failure path output could use the events collected thus far and (may) use it for generating this alternate output. Is there a way to do this?
Code for the flink job in the happy path was created and tested. It works as expected. I couldn't find anything in the documentation that shows a way to get the trigger's expiry condition to produce or set some state that signals to the reducer that it needs to generate alternate output?
Without seeing your code, I don't know if you're doing something odd, but normally you'd call .sideOutputLateData() on your stream (after calling .window())