ASP.NET Core Health Checks: Returning pre-evaluated results

Question

I'm evaluating the use of Microsoft Health Checks to improve routing of our internal load balancer. So far I'm very happy with the functionality provided by this feature and the community around it. However there's one thing I did not find yet and wanted to ask if it is possible out of the box:

The Health Checks seem to retrieve their own status as soon as they are requested. But because our service might have a hard time processing a lot of request in that given moment, the query to a thrid-party component like the SQL Server might take it's time to respond. Therefore, we would like to pre-evaluate that health check periodically (like every few seconds) and return that state when the health check api gets called.

The reason is, that we want our load balancer to get the health state as quickly as possible. Using pre-evaluated results seems to be good enough for our use case.

Now the question is: Is it possible to add a kind of "poll" or "auto-update" mechanism to the ASP.NET Core health checks? Or does this mean I have to implement my own health check returning values from a background service which pre-evaluates the results periodically?

Please note, I want to use pre-evaluated results on each request which is NOT HTTP Caching where the live result is cached for the next requests.

Answer 1

Short Version

This is already available and can already integrate with common monitoring systems. You may be able to tie Health Check directly into your monitoring infrastructure.

The details

The Health Check middleware covers this by periodically publishing metrics to a target, through any registered classes that implement the IHealthCheckPublisher.PublishAsync interface method.

services.AddSingleton<IHealthCheckPublisher, ReadinessPublisher>();

Publishing can be configured through HealthCheckPublisherOptions. The default period is 30 seconds. The options can be used to add delays, filter the checks to run etc:

services.Configure<HealthCheckPublisherOptions>(options =>
{
    options.Delay = TimeSpan.FromSeconds(2);
    options.Predicate = (check) => check.Tags.Contains("ready");
});

One option would be to cache the results (the HealthReport instance) with a publisher and serve them from another HealthCheck endpoint.

Perhaps a better option would be to push them to a monitoring system like Application Insights or a time-series database like Prometheus. The AspNetCore.Diagnostics.HealthCheck package provides a ton of ready-made checks and publishers for App Insights, Seq, Datadog and Prometheus.

Prometheus uses polling itself. It calls all its registered sources periodically to retrieve metrics. While that works for services, it won't work for eg CLI applications. For that reason, applications can push results to a Prometheus Gateway that caches the metrics until Prometheus itself requests them.

services.AddHealthChecks()
        .AddSqlServer(connectionString: Configuration["Data:ConnectionStrings:Sample"])
        .AddCheck<RandomHealthCheck>("random")
        .AddPrometheusGatewayPublisher();

Apart from pushing to Prometheus Gateway, the Prometheus publisher also offers an endpoint to retrieve live metrics directly, through the AspNetcore.HealthChecks.Publisher.Prometheus package. The same endpoint could be used by other applications to retrieve those metrics :

// default endpoint: /healthmetrics
app.UseHealthChecksPrometheusExporter();