Search code examples
websocketsignalrscaling

clients managment with multi SignalR servers behind a load balancer

Let's say I've got 2 Stock ticker servers that're pushing quotes to web browser clients. the 2 servers sits behind a load balancer (Round Robin mode). consider the following scenario:

  1. client A subscribe to Google stock on Server1 like so: Groups.Add(Context.ConnectionId, "Google");

  2. client B subscribe to Yahoo stock on Server2:Groups.Add(Context.ConnectionId, "Yahoo");

  3. client C subscribe to Google stock on Server2:Groups.Add(Context.ConnectionId, "Google");

Now both servers are already synced with the stock market so when a stock gets updated they both get the update at real time.

my question is:

when server2 push a new update like so:

Clients.Group("Google").tick(quote);

who are the clients it will send the message to? will it always be client C? I guess not, we have a load balancer in between so the connected clients at a given time may change, right? it may me C now, but next tick it can be clients A&C or only A. A web sockets's connection suppose to stay open so how does the load balancer will handle that, will it always forward the connection from 1 client to a specific server?

backplane won't help me here, because my 2 servers already synced and will send the same messages at the same time. so if I'll force them to route their messages through the backplane to the other server it will end up with duplicate messages to the clients like so:

server1 gets ticker X to Google at 10:00 --> route to backplane --> route to server 2 server2 gets ticker X to Google at 10:00--> route to backplane --> route to server 1

server 1 sends 2 X Google tickers to his clients server 2 sends 2 X Google tickers to his clients

Solution

  • OK, eventually I have synced all group subscriptions thorugh a shared cache (Redis) so all servers knows all users and their subsciptions. this way each server will know his current clients registerd groups, and will push the relevant data.

    Update:

    After much thought this is what we've ended up doing:

    1. load balancer will assign a sticky session to an incoming connection so each new connection will have a one constant SignalR server.
    2. section 1 will make the Redis sync redundant as each server will know all his clients.
    3. In case of a server\network failure, the SignalR client will reconnect and will be assigned with a new(in case of a server failure) server by the load balancer.
    4. After a reconnect the SignalR client will resubscribe with the relevant stocks(May be redundant if the failure was on the network and the load balancer redirect it to the old SignalR server, but I'll live with that).