Search code examples
.netazurerabbitmqazure-application-insightsdistributed-tracing

Azure Application Insights not showing the whole chain of events (in event-driven app), but all telemetry is there

Scenario

I have an containerized, event-driven, microservices app in ASP .NET Core just migrated to .NET 5.

All projects are using Microsoft.ApplicationInsights 2.16.0, and I have checked that all telemetry is sent and Sampling does not affect the results (at least individually for each micro service).

All projects publish and subscribe events to RabbitMQ.

I have my own rabbitmq nuget package to ease the access to RabbitMQ (which has been working properly for more than a year). I added this code to send telemetry identificators when an event is published:

var operation = _telemetryClient.StartOperation<DependencyTelemetry>($"Publish {theEvent.EventName.Replace(".", "-")}");
operation.Telemetry.Type = "Azure Service Bus"; // Just because I like the icon shown in Azure.
operation.Telemetry.Data = JsonConvert.SerializeObject(theEvent);
_basicProperties.Headers["x-telemetry-operationid"] = operation.Telemetry.Context.Operation.Id;
_basicProperties.Headers["x-telemetry-parentOperationId"] = operation.Telemetry.Id;
_channelProvider
     .GetChannel()
     .BasicPublish(_configuration.Exchange, theEvent.EventName, _basicProperties, Encoding.UTF8.GetBytes(payload));

I used this code to track the dependency when an event is received:

var operation = _telemetryClient.StartOperation<RequestTelemetry>($"Received {eventName.Replace(".", "-")}");
operation.Telemetry.Context.Operation.Id = basicDeliverEventArgs.BasicProperties.Headers["x-telemetry-operationId"];
operation.Telemetry.Context.Operation.ParentId = basicDeliverEventArgs.BasicProperties.Headers["x-telemetry-operationId"];
// Handle the message...
_telemetryClient.StopOperation(operation);

When a micro service receives a POST request, it will trigger an event. Several other micro services are subscribing to it, and they can also publish new events, which again will impact another set of microservices. And I would like to see it all in the End-to-end transaction details page when I click on the POST request.

But unfortunately this is not happening :(

I looked for the POST request in the Transaction Search: enter image description here

And clicked on it: enter image description here

You can see how "event-one" is published and how there is one micro services receiving it. But nothing else.

Now if I search for the event name, I find not only the publishing trace, but also more dependencies reflecting the events that the micro sevice published: enter image description here

If I click the one above the request, then I get a more comprehensive view of the whole transaction: enter image description here

But still, there is a lot of it missing here: All the SQL calls each micro service did, all the actions that microservices did when receiving events two, three and four.

And actually, all of it is available in Application Map: enter image description here

Which means that all telemetry is there correctly!

Questions

  1. Am I doing everything correctly?
  2. Is there a way to show the complete chain of events in the End-To-End?

Thanks in advance

Solution

  • Ok so I found it.

    Issue

    1. I was trying to tell AppInsights to do operations within my own little wrapper for the rabbitmq client library.
    2. While doing this, I was not passing the OperationId correctly (no parent needed).

    Looks like nowadays you should not use the StartOperation() method, but just TraceDependency() one anyway.

    Correct concept

    But actually the correct way to do this is to add instrumentation to the component you want to track. This instrumentation is done via System.Diagnostics namespace (Activities and DiagnosticListeners).

    Just to be clear, the RabbitMQ .NET client is the library that should be instrumented, so that other components in the framework can track what it is doing.

    Additionally, you will need IOBservers to tell Application Insights how to send those Activities to Azure.

    Fix

    1. I removed all the code related to Application Insights in my own wrapper.
    2. I downloaded the source code of Microsoft.Azure.ServiceBus.
    3. I downloaded the source code of Microsoft.ApplicationInsights.
    4. I added a RabbitMQDiagnosticListener class to my own wrapper, with methods StartSend, StartReceive and Stop. The code inside that class was practicaly a copy paste from the Microsoft.Azure.ServiceBus's own DiagnosticListeners.
    5. After some trial and error (to adjust the properties), I got it working and now the requests are showing all nested dependency calls correctly.

    As you can see I did not do the observers part. This is because I am basically replicating how Microsoft.Azure.ServicesBus is publishing Activities. Since Application Insights automatically tracks Microsoft.Azure.ServicesBus, it is taking care of my calls and in Azure it is actually shown as Azure ServiveBus even with the "Time spend in the queue" part.

    Git Repository: https://github.com/JoanComasFdz/dotnet-rabbitmq-client-instrumentation-app-insights

    Microsoft needs to improve a lot their own documentation, seems to be very incomplete and not updated.