I know that by design and out of the box a request and reply data processing is not possible with Flink. But consider for example a legacy TCP application, which opens a connection to a server and expects a response in tha same connection.
For example consider a legacy application, where the clients connect to a server via TCP and a custom protocol. They send some status information and expect a command as the response, where the command may depend on the current status.
Is it possible, to build a combined source,which inputs the TCP message into the processing, and sink, which recieves the processing result?
Building a source, which accepts TCP connections and creates events from messages seems straightforward, but getting the corresponding response to the corrent sink on the same worker(!) to send the response to the client see s tricky.
I know, that this can be implemented with an external component, but I'm wondering if this can be implemented directly in Flink with minimal overhead (e.g. for realtime performance reasons).
If this is possible, what would be the ways to do it and with which pros and cons?
Thank you!
Regards,
Kan
It depends how your server-processing pipeline looks like.
If the processing can be modeled as a single chain, as in Source -> Map/flatMap/filter -> Map/flatMap/filter -> ... -> sink, then you could pass the TCP connection itself the next operation together with the data (I supposed wrapped in a tuple or POJO). By virtue of being part of a chain it is guaranteed that the entire computation happens within a single worker.
But, the moment you do anything like grouping, windows etc. this is no longer possible, since the processing may continue on another worker.