We have a keyed process function that uses state and a "key by" being done immediately before that. The "key by" attribute involved transactional values and hence we expect many keys to be created. But these will be short-lived and we don't expect them to last for more than a day. Is there any way by which we can delete all the state associated with a key and the key itself manually from within the keyed process function?
Will simply setting the value of the associated state variables to null enable Flink to clean it up?
We are worried that even a very minimal amount of residual data that might be left back for every key-value would accumulate and contribute to huge state size.
One solution would be to configure state TTL so that the state is automatically deleted after some period of not being used. Or you can register a keyed timer in your keyed process function, and call clear() in the onTimer method to delete the state when the timer fires.