Lost metrics when using Spring Webflux Webclient with Jetty

Question

What I would like to achieve:

Get metrics when using Spring Webflux Webclient, but with Jetty HttpClient (instead of Netty)

What I tried:

    @Bean
    @Primary
    public WebClient getWebClient(final HttpClient httpClient) {
        return WebClient.builder()
                .baseUrl("http://some-host:8080/api/path")
                .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
                .clientConnector(new JettyClientHttpConnector(httpClient))
                .build();
    }
    @Bean
    public HttpClient getHttpClient(final MeterRegistry registry) {
        final var sslContextFactory = new SslContextFactory.Client();
        sslContextFactory.setSslContext(getCustomSSLContext());
        final var clientConnector = new ClientConnector();
        clientConnector.setSslContextFactory(sslContextFactory);
        final var httpClient = new HttpClient(new HttpClientTransportDynamic(clientConnector));
        return httpClient;
    }

with those dependencies:

<dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-elastic</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>io.opentelemetry</groupId>
            <artifactId>opentelemetry-exporter-otlp</artifactId>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-reactive-httpclient</artifactId>
        </dependency>

Issue:

Unfortunately, I am not getting any Webflux Webclient related metrics, nor Jetty related metrics.

Note:

Just to test, I did try using Netty as such:

  @Bean
    @Primary
    public WebClient getWebClient(ClientHttpConnector clientHttpConnector) {
        return WebClient.create("http://somehost:9200")
                .mutate()
                .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE, HttpHeaders.ACCEPT, MediaType.APPLICATION_JSON_VALUE)
                .clientConnector(clientHttpConnector)
                .build();
    }
    
    @Bean
    public ClientHttpConnector getClientHttpConnector(HttpClient httpClient) {
        return new ReactorClientHttpConnector(httpClient);
    }

    @Bean
    public HttpClient getHttpClient() {
        return HttpClient.create()
                .wiretap(true)
                .metrics(true, Function.identity())
                .protocol(HttpProtocol.HTTP11);
    }

And I do get some metrics, (question here, are they correct, or expected?)

reactor_netty_bytebuf_allocator_active_direct_memory
reactor_netty_bytebuf_allocator_active_heap_memory
reactor_netty_bytebuf_allocator_chunk_size
reactor_netty_bytebuf_allocator_direct_arenas
reactor_netty_bytebuf_allocator_heap_arenas
reactor_netty_bytebuf_allocator_normal_cache_size
reactor_netty_bytebuf_allocator_small_cache_size
reactor_netty_bytebuf_allocator_threadlocal_caches
reactor_netty_bytebuf_allocator_used_direct_memory
reactor_netty_bytebuf_allocator_used_heap_memory
reactor_netty_connection_provider_active_connections
reactor_netty_connection_provider_idle_connections
reactor_netty_connection_provider_max_connections
reactor_netty_connection_provider_max_pending_connections
reactor_netty_connection_provider_pending_connections
reactor_netty_connection_provider_total_connections
reactor_netty_http_client_address_resolver
reactor_netty_http_client_connect_time
reactor_netty_http_client_data_received
reactor_netty_http_client_data_received_time
reactor_netty_http_client_data_sent
reactor_netty_http_client_data_sent_time
reactor_netty_http_client_response_time
reactor_netty_http_client_tls_handshake_time

However, I would like to use Jetty instead, and sadly, not getting any metrics.

Question:

What is the correct way to get Jetty and/or Webflux Webclient specific metrics please?

Answer 1

None of the examples here use the WebClient instrumentation, but rather rely on the HTTP library instrumentation. In the case of Reactor Netty, the library is instrumented directly with Micrometer. This is not the case for the Jetty client.

Arguably, you should not need to do any of that, as WebClient itself is instrumented and provides metrics no matter which HTTP client library is used (Reactor Netty, Jetty, JDK).

As explained in the Spring Boot reference documentation, you should inject the WebClient.Builder instance that is auto-configured with your preferences already: JSON codecs, observability and more. In this case, the ObservationRegistry is automatically configured on the builder and you'll get the metrics and traces contributed by Spring Framework.

@Bean
public WebClient webClient(WebClient.Builder builder) {
    final var sslContextFactory = new SslContextFactory.Client();
    sslContextFactory.setSslContext(getCustomSSLContext());
    final var clientConnector = new ClientConnector();
    clientConnector.setSslContextFactory(sslContextFactory);
    final var httpClient = new HttpClient(new HttpClientTransportDynamic(clientConnector));

    return builder
            .baseUrl("http://some-host:8080/api/path")
            .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
            .clientConnector(new JettyClientHttpConnector(httpClient))
            .build();
}

If you would like lower-level metrics with the Jetty client, similar to what you get currently with Reactor Netty, please create an enhancement request on the Jetty project.