Redirecting client from load balancer to right consumer in Kafka

Question

I am working on a personal project in which I want to be able to send one message from a producer to an end-user. Each message will have a key that identifies the user that has to receive the message.
This is the overall structure I have imagined:

I cannot figure out how I can tell the load balancer that whenever a user with key 2 for example contacts the load balancer, then we have to set up a connection (possibly with a WebSocket) with the consumer handling partitions with key 2 in them. Probably something can be done by using the same technique Kafka uses whenever it has to assign a partition to the key, or by keeping track of the keys each consumer manages.

I do not know whether this is possible, but even if it was, the technique I described would probably make the code too coupled with the architecture.

Could you please help me out with how I can achieve this? I do not want to store messages on a remote data store and retrieve them from a random consumer. I want the consumer to be able to serve the user as soon as possible whenever a connection is established with it. If there is no connection with that user, then I can store the message and deliver it when the connection is ready.

Answer 1

I eventually found useful the Push Messaging technique they use at Netflix. The trick is to add another level of indirection, made of web servers. Whenever a new client connects to one of the web servers, a tuple <client_id, webserver_id> is saved in an external data store. When the consumer needs to send the message to the client having that specific key, it looks it up on the external registry to find where the client is connected. Once it finds it, it sends the message to the right web server that will push the message to the client.