How to export multiple versions of metrics with OpenTelemetry Prometheus exporter in dotnet?

Question

I’m currently integrating OpenTelemetry into a dotnet project and I’m using the Prometheus exporter for exposing my metrics on the /v1/metrics path. Given that this project is currently under development, we foresee major changes in the metrics that this system provides. We are looking for a way to export different versions of metrics on different endpoint paths simultaneously, such as /v2/metrics, /v3/metrics, etc.

This is the part of code in question:

    services.AddOpenTelemetry()
        .WithMetrics(opts => opts
            .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("rrfc"))
            .AddMeter(TelemetryUnit.CALL_TELEMETRY_METER)
            .AddPrometheusExporter(opt =>
            {
                opt.ScrapeEndpointPath = "/v1/metrics";
            }));
    
    app.UseOpenTelemetryPrometheusScrapingEndpoint();

Is there a way to configure multiple Prometheus exporters with different ScrapeEndpointPath options and use them in the same project? Or is there another approach to achieve this goal? Any help or guidance would be appreciated. Thanks in advance.

What I have already tried:

I have tried adding a second prometheus exporter with a different scraping endpoint but the last one will always be used.

Answer 1

As you observed, the last exporter configured will win. The doc comments on WithMetrics reads:

Note: This is safe to be called multiple times and by library authors. Each registered configuration action will be applied sequentially.

Instead of using the extension methods you would need to manually instantiate two separate MetricProviders, and scrape endpoints like so:

var meterProvider1 = Sdk.CreateMeterProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("rrfc"))
    .AddMeter(TelemetryUnit.CALL_TELEMETRY_METER)
    .AddPrometheusExporter("v1 metrics", opt =>
    {
        opt.ScrapeEndpointPath = "/v1/metrics";
    }, name: "v1 metrics");

var meterProvider2 = Sdk.CreateMeterProviderBuilder()
    .SetResourceBuilder(ResourceBuilder.CreateDefault().AddService("rrfc"))
    .AddMeter(TelemetryUnit.CALL_TELEMETRY_METER)
    .AddPrometheusExporter("v2 metrics", opt =>
    {
        opt.ScrapeEndpointPath = "/v2/metrics";
    });

app.UseOpenTelemetryPrometheusScrapingEndpoint(meterProvider1, null, null, null, "v1 metrics");
app.UseOpenTelemetryPrometheusScrapingEndpoint(meterProvider2, null, null, null, "v2 metrics");

Looking at the implementation, passing null to the path parameter will cause a lookup against the exporter options to retrieve the configured exporter path when using named options.

Manually making multiple instances of MeterProvider means figuring out how to manage their lifecycle. The WithMetrics extension method does this by hooking in to the .NET dependency injection framework.

Per the doc comments on CreateMeterProviderBuilder state:

Creates a MeterProviderBuilder which is used to build a MeterProvider. In a typical application, a single MeterProvider is created at application startup and disposed at application shutdown. It is important to ensure that the provider is not disposed too early.

A quick solution is registering both MeterProvider instances as singletons, like so:

services.AddSingleton(meterProvider1);
services.AddSingleton(meterProvider2);

This should properly dispose of both instances when the application is terminated.