Request Id isn't present in logs after going to kafka

Question

My service is using spring boot version 3.1.0 and my use case is only the presence of request Id in logs so that any requests coming from other services so that they can be tracked with the request id they bring or the one generated at our side. This will help me see an entire flow of a user easily for debugging. I have added the following micrometer dependencies to a spring boot application(its a large scale application which can get 20k+ requests per minute)

    <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-observation</artifactId>
        </dependency>
        <dependency>
            <groupId>io.micrometer</groupId>
            <artifactId>micrometer-tracing-bridge-brave</artifactId>
        </dependency>

I receive a request on my controller and after processing it I am using default kafkaTemplate to send the message via kafka

@Autowired
private KafkaTemplate<String, UnifiedMessageRequest> kafkaTemplate;

and

kafkaTemplate.send(topicName, unifiedMessageReqSaved);

Consumer Side Code:

@KafkaListener(topics = "${spring.kafka.template.topic-p0}",
            groupId = "${spring.kafka.consumer.group-id-p0}",
            concurrency = "${spring.kafka.consumer.p0-concurrency}")
    public void consumeP0(final UnifiedMessageRequest message, final Acknowledgment acknowledgment) {
        sendAndAcknowledge(message, acknowledgment);
    }

Spring kafka properties

   spring:
    kafka:
     exponential-retries:
      maxRetries: 3
      multiplier: 2
      initialInterval: 1000
      maxInterval: 300000
      skippedExceptions:
        - com.example.exceptions.UnifiedValidationException
        - jakarta.validation.ValidationException
     bootstrap-servers: localhost:9092
     template:
      topic-p0: p0
     producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      properties:
        linger.ms: 100
        enable.idempotence: true
        spring.json.add.type.headers: true
     consumer:
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      group-id-p0: P0_LOCALDEV
      enable-auto-commit: false
      auto-offset-reset: latest
      p0-concurrency: 10
      properties:
        isolation.level: read_committed
        spring.json.type.mapping: Event:com.example.dto.request.UnifiedMessageRequest
        spring.json.value.default.type: com.example.dto.request.UnifiedMessageRequest
        spring.json.trusted.packages: com.example.common.model,com.example.dto.request
        spring.json.use.type.headers: false

Logs before sending message to kafka:

spring-boot-3-service|[30m2023-10-11 15:50:26,766[0;39m [34mRequestID:[da45cb98-cf8c-4774-82dc-87e8d8b69d2e][0;39m-[172.26.3.223] [XNIO-1 task-2] APP_CONTEXT:[] [34mINFO [0;39m [[34mXNIO-1 task-2[0;39m] [33mc.j.f.n.n.s.i.SomeServiceImpl[0;39m: Example log present with request ID

Logs after message getting consumed:

spring-boot-3-service|[30m2023-10-11 19:15:53,165[0;39m [1;31mRequestID:[][0;39m-[] [scheduling-1] APP_CONTEXT:[] [1;31mERROR[0;39m [[34mscheduling-1[0;39m] [33mc.j.f.n.n.r.TemplateManagerService[0;39m: Log Example present without Request Id

I have simplified the code. I want to make least amount of changes to enable request ID in logs. I am new to spring boot and will be great if someone could direct me towards any property or config that would help me pass the request trace id.

Answer 1

3.1.4 is the current 3.1.x version.

Spring for Apache Kafka observation is not enabled by default; there is an open issue to add Boot properties to enable it: https://github.com/spring-projects/spring-boot/issues/36452

In the meantime, you have to set observationEnabled on the KafkaTemplate and on listener containers' ContainerProperties (or on the factory that creates the containers).

https://docs.spring.io/spring-kafka/docs/current/reference/html/#observation