Spring Boot 3.x Observability with Micrometer & Datadog for HTTP services and Kafka Consumer

Question

I am trying to utilise the Observability API from Spring Boot 3.x in my application for tracing and metrics but I'm confused with the necessary setup on how to get proper traceability and metrics details.

I have a create a sample Spring Boot application for testing.

I have added these dependencies in the pom.xml:

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-aop</artifactId>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-datadog</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-tracing-bridge-brave</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-tracing</artifactId>
    </dependency>
</dependencies>

application.yml:

spring:
  application:
    name: datadog-sample

server:
  port: 8090

management:
  metrics:
    distribution:
      percentiles-histogram:
        http:
          server:
            requests: true
  endpoint:
    health:
      cache:
        time-to-live: 6s
      show-details: always
    metrics:
      enabled: true
  endpoints:
    web:
      exposure:
        include: health,info,metrics
  health:
  jmx:
    metrics:
      export:
        enabled: true
        step: 1m
  info:
    env:
      enabled: true
  datadog:
    metrics:
      export:
        apiKey: test
  tracing:
    sampling:
      probability: 1.0
    propagation:
      type: W3C

logging:
  pattern:
    console: .%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]
      - %msg%n

TestController:

@RestController
@Slf4j
public class TestController {
    @GetMapping(value = "/method1")
    public ResponseEntity<String> method1(@RequestParam String input) throws IOException, InterruptedException {
        log.info("Inside the method1 with data = {}",input);
        HttpRequest request = HttpRequest.newBuilder().uri(URI.create("http://localhost:8090/method2")).build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        return ResponseEntity.ok(response.body());
    }

    @GetMapping(value = "/method2")
    public ResponseEntity<String> method2() {
        log.info("Inside the method2");
        return ResponseEntity.ok("Called method2 successfully");
    }
}

Problem: When Service 1 is invoked (http://localhost:8090/method1?input=testdata), it internally calls Service 2 and generating Trace Id and Span Id but for each service, it's generating different Trace Ids given below in the log:

. INFO [datadog-sample,652553b7e89ee89b58c1c37b35cb6102,58c1c37b35cb6102] - Inside the method1 with data = testdata
. INFO [datadog-sample,652553b7ec4d43c0f0e090c94225d91c,f0e090c94225d91c] - Inside the method2

Questions:

• Shouldn't this be a single Trace Id with multiple Span Id so that the flow can be traced easily?

• Should I need to use @Obesrved annotation anywhere so that I don't need to customise any behaviour?

• To send metrics/observability details to Datadog:

  1. Do I need to add anything specific in the code/configuration apart from including Datadog specific dependencies in the POM along with running the Datadog agent in the background?

• Does micrometer observability works out of the box for spring cloud-kafka-binder application or do it needs any specific configuration? If yes, can someone provide a reference example?

Answer 1

I found the issue in your code on why you were getting different traceIds.

Shouldn't this be single TraceId with multiple SpanId, so that the flow can be traced easily. Should I need to use @Observed annotation anywhere as I don't want to customise any behaviour?

Answer: Tracing Context Propagation with Micrometer Tracing via java.net.http.HttpRequest/java.net.http.HttpResponse is not working as tested.

So, I guess it was not considering the call to method2 as part of the same tracing context propagation although it should have been.

---

Please use RestTemplate way to send the request and for getting the response. The propagation of trace will work for sure with RestTemplate.

Read the docs here: https://spring.io/blog/2022/10/12/observability-with-spring-boot-3

You will need to add OpenTelemetry (io.micrometer:micrometer-tracing-bridge-otel) dependency for Tracing Context Propagation with Micrometer Tracing (needed for RestTemplate) in the pom.xml.

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>

For Micrometer Integration with Datadog, you will need to add io.micrometer:micrometer-registry-datadog dependency in the pom.xml:

    <dependency>
        <groupId>io.micrometer</groupId>
        <artifactId>micrometer-registry-datadog</artifactId>
        <scope>runtime</scope>
    </dependency>

The pom.xml that I have used:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.1.4</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example</groupId>
    <artifactId>spring-boot-datadog</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>spring-boot-datadog</name>
    <description>Demo project for Spring Boot</description>
    <properties>
        <java.version>17</java.version>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-otel</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <scope>runtime</scope>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-registry-datadog</artifactId>
            <scope>runtime</scope>
        </dependency>
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <excludes>
                        <exclude>
                            <groupId>org.projectlombok</groupId>
                            <artifactId>lombok</artifactId>
                        </exclude>
                    </excludes>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Custom Configuration class:

import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.RestTemplate;

import io.micrometer.observation.ObservationRegistry;
import io.micrometer.observation.aop.ObservedAspect;

@Configuration
public class ExampleConfiguration {
    
    @Bean
    ObservedAspect observedAspect(ObservationRegistry observationRegistry) {
        return new ObservedAspect(observationRegistry);
    }

    @Bean
    RestTemplate restTemplate(RestTemplateBuilder builder) {
        return builder.build();
    }
    
}

You have to create ObservedAspect bean to enable Observability via AOP.

Update the TestController to this :

import java.net.URI;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

import io.micrometer.observation.annotation.Observed;
import lombok.extern.slf4j.Slf4j;

@RestController
@Slf4j
public class TestController {
    

    @Autowired
    private RestTemplate restTemplate;
    
    @Observed(name = "method1", contextualName = "method1")
    @GetMapping(value = "/method1")
    public ResponseEntity<String> method1(@RequestParam String input) {
        log.info("Inside the method1 with data = {}", input);
        ResponseEntity<String> response = restTemplate.getForEntity(URI.create("http://localhost:8090/method3"),
                String.class);
        return ResponseEntity.ok(response.getBody());
    }

    @Observed(name = "method2", contextualName = "method2")
    @GetMapping(value = "/method3")
    public ResponseEntity<String> method2() {
        log.info("Inside the method2");
        return ResponseEntity.ok("Called method2 successfully");
    }
} 

Note: You will need to use @Observed to put data into Datadog metrics.

application.properties:

spring.application.name=datadog-sample
server.port=8090

management.datadog.metrics.export.api-key=2196881cdea46553c0e65ab7aa8af0de
management.datadog.metrics.export.application-key=8df3b2e284a626127f5b05a4f40fde064036fcef
management.datadog.metrics.export.uri=https://us5.datadoghq.com

management.metrics.distribution.percentiles-histogram.http.server.requests=true

management.endpoints.web.exposure.include=health,info,metrics

management.tracing.sampling.probability=1.0
management.tracing.propagation.type=w3c

management.info.env.enabled=true

management.jmx.metrics.export.enabled=true
management.jmx.metrics.export.step=1m

management.endpoint.health.cache.time-to-live=6s
management.endpoint.health.show-details=always
management.endpoint.metrics.enabled=true

logging.pattern.console=.%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}] - %msg%n

For datadog, you will need to set these properties to start using micrometer-registry-datadog with spring-boot:

• management.datadog.metrics.export.api-key
• management.datadog.metrics.export.application-key
• management.datadog.metrics.export.uri

Steps on how to setup Datadog:

1. Go to this url -> https://us5.datadoghq.com/
2. Click on Try free and sign up for 14 days trial with giving your credit card.
3. Enter the required details and click on sign up.
4. It will ask you install Data Dog Agent 7 initially so that it will detect the agent and let you proceed. Otherwise, it doesn't let you proceed. So, install the Datadog agent.

For macOS, I was able to install agent 7 via this command:

DD_AGENT_MAJOR_VERSION=7 DD_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX DD_SITE="us5.datadoghq.com" bash -c "$(curl -L https://s3.amazonaws.com/dd-agent/scripts/install_mac_os.sh)"

This agent runs in the background.

To verify that datadog agent is running. Enter this command -

datadog-agent status

5. Later on, you don't need to run that agent.
6. Go to your Datadog Dashboard, on the left-hand side below, there will be a profile icon with email id in it. Click on that -> Click on the Organization Settings.

7. To get api-key, see the screenshot given below.

8. Click on copy key to copy the api key.

9. Same process to get the application-key, see the screenshot given below.

10. Click on copy key to copy the application key.

10. For trial, by default, they give the following region server url as https://us5.datadoghq.com/. You can copy from browser address itself.

That's all.

---

When I hit the request, I was able to get the same trace Id with multiple span ids in the console.

Output:

Spring logs with same traceIds:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
[32m :: Spring Boot :: [39m              [2m (v3.1.4)[0;39m

. INFO [datadog-sample,,] - Starting SpringBootDatadogApplication using Java 17.0.1 with PID 19939 (/Users/anish/Downloads/spring-boot-datadog/target/classes started by anish in /Users/anish/Downloads/spring-boot-datadog)
. INFO [datadog-sample,,] - No active profile set, falling back to 1 default profile: "default"
. INFO [datadog-sample,,] - Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable
. INFO [datadog-sample,,] - For additional web related logging consider setting the 'logging.level.web' property to 'DEBUG'
. INFO [datadog-sample,,] - Tomcat initialized with port(s): 8090 (http)
. INFO [datadog-sample,,] - Starting service [Tomcat]
. INFO [datadog-sample,,] - Starting Servlet engine: [Apache Tomcat/10.1.13]
. INFO [datadog-sample,,] - Initializing Spring embedded WebApplicationContext
. INFO [datadog-sample,,] - Root WebApplicationContext: initialization completed in 1448 ms
. INFO [datadog-sample,,] - publishing metrics for DatadogMeterRegistry every 1m
. INFO [datadog-sample,,] - LiveReload server is running on port 35729
. INFO [datadog-sample,,] - Exposing 3 endpoint(s) beneath base path '/actuator'
. INFO [datadog-sample,,] - Tomcat started on port(s): 8090 (http) with context path ''
. INFO [datadog-sample,,] - Started SpringBootDatadogApplication in 2.738 seconds (process running for 3.802)
. INFO [datadog-sample,,] - Initializing Spring DispatcherServlet 'dispatcherServlet'
. INFO [datadog-sample,,] - Initializing Servlet 'dispatcherServlet'
. INFO [datadog-sample,,] - Completed initialization in 1 ms
. INFO [datadog-sample,666c0ee462f732fe199562808d56e40b,75defc5f90043736] - Inside the method1 with data = testdata
. INFO [datadog-sample,666c0ee462f732fe199562808d56e40b,95b4fc1dda6635e9] - Inside the method2

Finally, I was able to make it working. :)

Screenshot to verify that it has picked up the metrics:

Go to Metrics -> Explorer menu on the Datadog dashboard.

Does micrometer observability works out of the box for springcloud-kafka-binder application or it needs any specific configuration? Any reference example for this?

Yes. Doc link is given below. Please refer to this:

• https://docs.spring.io/spring-kafka/reference/appendix/micrometer.html