ServiceBusClient SendMessageAsync very slow

Question

I have the following method that I'm debugging in vscode:

public async Task PublishReport(string correlationId)
    {
        _methodName = nameof(GetReport);
        _correlationId = correlationId;
        LogInfo("Started");

        DateTime now = DateTime.Now;
        DateTime lastPoll = await GetTimeOfLastPoll(_correlationId);

        List<AmxGetReportResponse> reportRecs = await GetReport(lastPoll, _correlationId);
        var credential = new ChainedTokenCredential(
                                new ManagedIdentityCredential(),
                                new VisualStudioCredential(),
                                new AzureCliCredential()
                                );
        LogInfo($"Now logged into azure");
        var serviceBusClient = new ServiceBusClient($"{_amxOptions.ServiceBusNamespace}.servicebus.windows.net", credential);
        LogInfo($"Have service bus client");
        var topicSendClient = serviceBusClient.CreateSender(_amxOptions.ServiceBusTopicName);
        LogInfo($"Have topic sender");

        foreach (var reportRec in reportRecs)
        {
            string msg = _amxReport2CatsUpdate.Execute(reportRec);
            LogInfo($"Mapped msg={msg}");
            var serviceBusMsg = new ServiceBusMessage(msg);
            LogInfo($"Have service bus message");
            await topicSendClient.SendMessageAsync(serviceBusMsg);
            LogInfo($"Sent msg to sb topic {_amxOptions.ServiceBusTopicName}: {msg}");
        }

        await SetTimeOfLastPoll(now, _correlationId);
        LogInfo("Leaving");
    }

It does work and will publish a message for each record of the report. The total size of the report xml is 30k and each record is approx 1k.

My problem is, it takes roughly 1 min 45 seconds for the SendMessageAsync method to run?

Answer 1

Service Bus clients are intended to be long-lived and, ideally, treated as singletons. (docs)

The pattern that you're using here requires that a new connection be created, authentication be performed, a link opened, and authorization to be performed for each call to PublishReport. Because all of that happens lazily when the first service operation is requested, your first call to SendMessagesAsync is paying the cost.

I suspect that the majority of the time is being spent attempting to acquire a credential. I'd suggest collecting Azure.Identity logs to help understand what credential is being selected and whether failures/retries are taking place. The approach described in that guide will also capture Service Bus logs, which emit an entry when sending starts and completes which would allow you to calculate how long the actual service operation took. Specifically, you'll be looking for events 1, 2, and 3, from the "Azure-Messaging-ServiceBus" event source.

It is also worth noting that the Service Bus clients are not disposed by PublishReport, which will orphan connections until they timeout, potentially causing delays due to contention in the network stack.