Hangfire job filter for Application Insights should log only when job retries are exhausted and the job moves to Failed state

Question

I'm using the following Hangfire job filter to log job executions in Application Insights. My goal is to create alerts for jobs that have ultimately failed after all retries are exhausted. However, I'm encountering an issue where Application Insights logs every exception thrown, even when Hangfire is going to retry the job. This results in premature alerts, as I only want to log when a job enters the "Failed" tab in Hangfire, meaning it has stopped retrying and has truly failed.

Here is the HangfireApplicationInsightsFilter class I'm using to log the jobs:

public sealed class HangfireApplicationInsightsFilter : IServerFilter
{
    private readonly TelemetryClient _telemetryClient;

    public HangfireApplicationInsightsFilter(TelemetryClient telemetryClient) => _telemetryClient = telemetryClient;

    public void OnPerforming(PerformingContext filterContext)
    {
        var operation = _telemetryClient.StartOperation<RequestTelemetry>(GetJobName(filterContext.BackgroundJob));
        operation.Telemetry.Properties.Add("JobId", filterContext.BackgroundJob.Id);
        operation.Telemetry.Properties.Add("Arguments", GetJobArguments(filterContext.BackgroundJob));

        filterContext.Items["ApplicationInsightsOperation"] = operation;
    }

    public void OnPerformed(PerformedContext filterContext)
    {
        if (filterContext.Items["ApplicationInsightsOperation"] is not IOperationHolder<RequestTelemetry> operation) return;

        if (filterContext.Exception == null || filterContext.ExceptionHandled)
        {
            operation.Telemetry.Success = true;
            operation.Telemetry.ResponseCode = "Success";
        }
        else
        {
            operation.Telemetry.Success = false;
            operation.Telemetry.ResponseCode = "Failed";

            var operationId = operation.Telemetry.Context.Operation.Id;

            var exceptionTelemetry = new ExceptionTelemetry(filterContext.Exception);
            exceptionTelemetry.Context.Operation.Id = operationId;
            exceptionTelemetry.Context.Operation.ParentId = operationId;

            _telemetryClient.TrackException(exceptionTelemetry);
        }

        _telemetryClient.StopOperation(operation);
    }

    private static string GetJobName(BackgroundJob backgroundJob) => $"{backgroundJob.Job.Type.Name}.{backgroundJob.Job.Method.Name}";

    private static string GetJobArguments(BackgroundJob backgroundJob) => JsonSerializer.Serialize(backgroundJob.Job.Args);
}

This filter logs every exception, even when the job is retried. What I need is for the exception to be logged only after the final retry attempt has failed.

How can I modify my `HangfireApplicationInsightsFilter to log exceptions only when Hangfire has stopped retrying the job, and the job is moved to the "Failed" state?

Any guidance or insights on how to achieve this behavior would be greatly appreciated.

Answer 1

You can create a filter that implements the IApplyStateFilter interface like the below :

public sealed class HangfireApplicationInsightsFilter : IServerFilter, IApplyStateFilter
{
 // .. the other implementations
 public void OnStateApplied(ApplyStateContext context, IWriteOnlyTransaction transaction)
 {
     Logger.DebugFormat(
         "Job `{0}` state was changed from `{1}` to `{2}`",
         context.BackgroundJob.Id,
         context.OldStateName,
         context.NewState.Name);

   if (context.NewState is FailedState failed)
   {
     // do something with failed state.
   }
  }
}