How do I set up Azure alerts for failed Hangfire jobs stored in Azure Cache for Redis?

Question

I'm working on a project that schedules Hangfire jobs. The jobs are stored in an Azure Redis store (Azure Cache for Redis).

Here is the setup for clarification:

services.AddHangfire(x => x
    .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
    .UseRecommendedSerializerSettings()
    .UseRedisStorage(redisConnectionString, redisStorageOptions)
    .UseBatches()
    .UseThrottling(ThrottlingAction.RetryJob, 1.Seconds())
    .UseTagsWithRedis(new() { TagsListStyle = TagsListStyle.Dropdown }, redisStorageOptions));

services.AddHangfireServer((provider, options) =>
{
    options.Activator = new HangfireJobActivator(provider);
    options.WorkerCount = Environment.ProcessorCount * 5;
    options.HeartbeatInterval = 10.Seconds();
    options.SchedulePollingInterval = 1.Seconds();
    options.Queues = new[] { "critical", "default" };
    options.ServerName = Environment.MachineName;
});

I need to set up an alert for failed Hangfire jobs. I opened the Redis database in RedisInsights and here is what I got:

To set up the alert, I navigated to: Azure Portal > Azure Cache for Redis > [Resource Name] > Alerts > Create alert rule > Custom log search.

Now I'm uncertain about how to access the specific data within the Redis cache to monitor failed jobs and set up the corresponding alert in Azure.

Here is what I get from Azure KQL's IntelliSense:

If this is not possible then we need a way to send the failed Hangfire job data to Azure and one way would be through log messages based on which we can trigger alerts.

Answer 1

Thanks to @akseli for pointing out that approach!

Here's the filter for my own use case:

public sealed class HangfireApplicationInsightsFilter : IServerFilter
{
    private readonly TelemetryClient _telemetryClient;

    public HangfireApplicationInsightsFilter(TelemetryClient telemetryClient) => _telemetryClient = telemetryClient;

    public void OnPerforming(PerformingContext filterContext)
    {
        var operation = _telemetryClient.StartOperation<RequestTelemetry>(GetJobName(filterContext.BackgroundJob));
        operation.Telemetry.Properties.Add("JobId", filterContext.BackgroundJob.Id);
        operation.Telemetry.Properties.Add("Arguments", GetJobArguments(filterContext.BackgroundJob));

        filterContext.Items["ApplicationInsightsOperation"] = operation;
    }

    public void OnPerformed(PerformedContext filterContext)
    {
        if (filterContext.Items["ApplicationInsightsOperation"] is not IOperationHolder<RequestTelemetry> operation) return;

        if (filterContext.Exception == null || filterContext.ExceptionHandled)
        {
            operation.Telemetry.Success = true;
            operation.Telemetry.ResponseCode = "Success";
        }
        else
        {
            operation.Telemetry.Success = false;
            operation.Telemetry.ResponseCode = "Failed";

            var operationId = operation.Telemetry.Context.Operation.Id;

            var exceptionTelemetry = new ExceptionTelemetry(filterContext.Exception);
            exceptionTelemetry.Context.Operation.Id = operationId;
            exceptionTelemetry.Context.Operation.ParentId = operationId;

            _telemetryClient.TrackException(exceptionTelemetry);
        }

        _telemetryClient.StopOperation(operation);
    }

    private static string GetJobName(BackgroundJob backgroundJob) => $"{backgroundJob.Job.Type.Name}.{backgroundJob.Job.Method.Name}";

    private static string GetJobArguments(BackgroundJob backgroundJob) => JsonSerializer.Serialize(backgroundJob.Job.Args);
}

I've packaged this solution into a NuGet package. Hope it helps someone else trying to do the same. You can find more details and instructions in the GitHub repo.