.Net Core SignalR Connection between services has an issue

Question

I have several services communicating with each other via SignalR connection. They are both standard .net core web api services, let's say I have service A and service B.

Service A has SignalR hub configured and accepting connections, Service B is connected to it using HubConnection class from Microsoft.AspNetCore.SignalR.Client package. And is listening to some messages from service A with: hubConnection.On('...')

Service B also has a signalR Hub and is accepting connections from website clients.

Service A is sending about 2000 messages in a minute to service B, and service B also streams these messages to all the connected clients. there are about 80 client connections to service B at the same time.

The issue is that the messages from Service A are arriving very late, they are about 2 minutes late. I am logging the line when the message is sent from service A and is received in service B and it's about 2 minute difference. I don't have any ideas why this happens, 2k messages in a minute isn't that much as I Believe.

here's a diagram:

Service B does this type of logic on each received message:

hubConnection.On(method.Name, parameterTypes, @params =>
{
    return Task.Run(() =>
    {
        var activity = new Activity("HubListener");
        try
        {
            activity.Start();
            activity.AddTag("TraceId", Guid.NewGuid().ToString());
            activity.AddTag("ArrivalTime", DateTime.UtcNow);

            var obj = _serviceProvider.GetRequiredService(method.DeclaringType)
            method.Invoke(obj, @params);
        }
        catch (Exception e)
        {
            Log.Error(e, "Error In Hub Listener");

        }
        finally
        {
            activity.Stop();
        }
    });
});

the method is taken via reflection from my BroadcastService class (there are other methods too that's why use reflection to subscribe dynamically):

public class BroadcastService {
        private readonly IHubContext<BaseHub> _baseHubContext;

        public BroadcastService(IHubContext<BaseHub> baseHubContext)
        {
            _baseHubContext = baseHubContext ?? BaseHub.CurrentContext;
        }

        public void BroadcastToClients(Message messageFromServiceA)
        {
             // it takes about 2 minutes for message to arrive at this place after it is sent from Service A.
             // clients are separated in groups
             foreach(var group in clientGroups){
          _baseHubContext
             .Clients
             .Group(group.Name)
             .SendAsync("onMessage",messageFromServiceA)
          }
        }
}

If I run service B on local machine and there are no connections from clients the messages are not delayed, this has something to do with number of client connections but 80 is not a lot. I don't know if the messages on hubConnection are executed sequentially, but this should not be an issue because I use Task.Run and this should run handler asynchronously.

I would appreciate any kind of help, I will edit the question if more details are required.

Answer 1

I finally found answer to my question, the issue was that SignalR client (as well as server) processes messages sequentially, I understood this by logging start and end timestamps of the handlers. After some research I found out that this was by design in SignalR Core, here's similar question and answer.

To fix my issue I just run task in background and don't wait for it:

old code:

hubConnection.On(method.Name, parameterTypes, @params =>
{
    return Task.Run(() =>
    {
        ...
    });
});

new code:

hubConnection.On(method.Name, parameterTypes, @params =>
{
    _ = Task.Run(() =>
    {
        ...
    });

    return Task.CompletedTask;
});

Now it works just fine.