Logging Hangfire jobs to Application Insights and correlating activity to an Operation Id

Question

I feel like this should be a lot simpler than it's turning out to be, or I am just over thinking it too much.

I have a .NET Core 3.1 Web API application, which is using HangFire to process some jobs in the background. I have also configured Application Insights to log Telemetry from the .NET Core API.

I can see logging events and dependency telemetry data logged in Application Insights. However, each event/log/dependency is recorded against a unique OperationId and Parent Id.

I am trying to determine how to ensure that any activity which is logged, or any dependencies which are used in the context of the background job are logged against the OperationId and/or Parent Id of the original request which queued the background job.

When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job. When the job is then performed, I can get back that OperationId. What I then need to do is make that OperationID available throughout the context/lifetime of the job execution, so that it is attached to any Telemetry sent to Application Insightd.

I thought I could create a IJobContext interface, which could be injected into the class which performs the job. Within that context I could push the OperationID. I could then create a ITelemetryInitializer which would also take the IJobContext as a dependency. In the ITelemetryInitializer I could then set the OperationID and ParentId of the telemetry being sent to Application Insights. Here's some simple code:

public class HangFirePanelMessageQueue : IMessageQueue
{
    private readonly MessageProcessor _messageProcessor;
    private readonly IHangFireJobContext _jobContext;
    private readonly TelemetryClient _telemetryClient;

    public HangFirePanelMessageQueue(MessageProcessor panelMessageProcessor,
        IIoTMessageSerializer iotHubMessageSerialiser,
        IHangFireJobContext jobContext, TelemetryClient telemetryClient)
    {
        _messageProcessor = panelMessageProcessor;
        _jobContext = jobContext;
        _telemetryClient = telemetryClient;
    }

    public async Task ProcessQueuedMessage(string message, string operationId)
    {
        var iotMessage = _iotHubMessageSerialiser.GetMessage(message);

        _jobContext?.Set(iotMessage.CorrelationID, iotMessage.MessageID);

        await _messageProcessor.ProcessMessage(iotMessage);
    }

    public Task QueueMessageForProcessing(string message)
    {
        var dummyTrace = new TraceTelemetry("Queuing message for processing", SeverityLevel.Information);
        _telemetryClient.TrackTrace(dummyTrace);
        string opId = dummyTrace.Context.Operation.Id;

        BackgroundJob.Enqueue(() =>
        ProcessQueuedMessage(message, opId));

        return Task.CompletedTask;
    }
}

The IJobContext would look something like this:

public interface IHangFireJobContext
{
    bool Initialised { get; }

    string OperationId { get; }

    string JobId { get; }

    void Set(string operationId, string jobId);
}

And then I would have an ITelemetryInitializer which enriches any ITelemetry:

public class EnrichBackgroundJobTelemetry : ITelemetryInitializer
{
    private readonly IHangFireJobContext jobContext;

    public EnrichBackgroundJobTelemetry(IHangFireJobContext jobContext)
    {
        this.jobContext = jobContext;
    }

    public void Initialize(ITelemetry telemetry)
    {
        if (!jobContext.Initialised)
        {
            return;
        }

        telemetry.Context.Operation.Id = jobContext.OperationId;
    }
}

The problem I have however is that the ITelemetryInitializer is a singleton, and so it would be instantiated once with a IHangFireJobContext which would then never update for any subsequent HangFire job.

I did find the https://github.com/skwasjer/Hangfire.Correlate project, which extends https://github.com/skwasjer/Correlate. Correlate creates a correlation context which can be accessed via a ICorrelationContextAccessor which is similar to the IHttpContextAccessor.

However, the footnotes for Correlate state "Please consider that .NET Core 3 now has built-in support for W3C TraceContext (blog) and that there are other distributed tracing libraries with more functionality than Correlate." which lists Application Insights as one of the alternatives for more Advanced distributed tracing.

So can anyone help me understand how I can enrich any Telemetry going to Application Insights when it is created within the context of a HangFire job? I feel the correct answer is to use an ITelemetryInitializer and populate the OperationId on that ITelemetry item, however, I am not sure what dependancy to inject into the ITelemetryInitialzer in order to get access to the HangFire Job Context.

Answer 1

When I queue a job, I can get the current OperationId of the incoming HTTP request, and I push that into the HangFire queue with the job.

So, am I correct to say that you have a controller action that pushes work to hangfire? If so What you can do is inside the controller method get the operation id and pass it to the job. Use that operation id to start a new operation using the operation Id. That operation, together with all the telemetry generated during that operation, will be linked to the original request.

I have no hangfire integration but the code below shows the general idea: some work is queued to be done in the background and should be linked to the request regarding the telemetry:

        [HttpGet("/api/demo5")]
        public ActionResult TrackWorker()
        {
            var requestTelemetry = HttpContext.Features.Get<RequestTelemetry>();

            _taskQueue.QueueBackgroundWorkItem(async ct =>
            {
                using(var op = _telemetryClient.StartOperation<DependencyTelemetry>("QueuedWork", requestTelemetry.Context.Operation.Id))
                {
                    _ = await new HttpClient().GetStringAsync("http://blank.org");

                    await Task.Delay(250);
                    op.Telemetry.ResultCode = "200";
                    op.Telemetry.Success = true;
                }
            });

            return Accepted();
        }

The full example can be found here.