Search code examples
spring-bootloggingslf4jopentracingjaeger

How to enrich Jaeger opentracing data with the application logs (produced by slf4j) for Spring Boot?

There is an existing Spring Boot app which is using SLF4J logger. I decided to add the support of distributed tracing via standard opentracing API with Jaeger as the tracer. It is really amazing how easy the initial setup is - all that is required is just adding two dependencies to the pom.xml:

    <dependency>
        <groupId>io.opentracing.contrib</groupId>
        <artifactId>opentracing-spring-web-autoconfigure</artifactId>
        <version>${io.opentracing.version}</version>
    </dependency>

    <dependency>
        <groupId>io.jaegertracing</groupId>
        <artifactId>jaeger-core</artifactId>
        <version>${jaegerVersion}</version>
    </dependency>

and providing the Tracer bean with the configuration:

@Bean
public io.opentracing.Tracer getTracer() throws ConfigurationException {
    return new new io.jaegertracing.Tracer.Builder("my-spring-boot-app").build();
}

All works like a charm - app requests are processed by Jaeger and spans are created:

enter image description here

However, in the span Logs there are only preHandle & afterCompletion events with info about the class / method that were called during request execution (no logs produced by slf4j logger are collected) :

enter image description here

The question is if it is possible to configure the Tracer to pickup the logs produced by the app logger (slf4j in my case) so that all the application logs done via: LOG.info / LOG.warn / LOG.error etc. would be also reflected in Jaeger

NOTE: I have figured out how to log to span manually via opentracing API e.g.:

Scope scope = tracer.scopeManager().active();
if (scope != null) {
    scope.span().log("...");
}

And do some manual manipulations with the ERROR tag for exception processing in filters e.g. 

} catch(Exception ex) {
    Tags.ERROR.set(span, true);
    span.log(Map.of(Fields.EVENT, "error", Fields.ERROR_OBJECT, ex, Fields.MESSAGE, ex.getMessage()));
    throw ex
}

But, I'm still wondering if it is possible to configure the tracer to pickup the application logs automatically:

  • LOG.info -> tracer adds new log to the active span
  • LOG.error -> tracer adds new log to the active span plus adds ERROR tag

UPDATE: I was able to add the application logs to the tracer by adding wrapper for the logger e.g.

public void error(String message, Exception e) {
    Scope scope = tracer.scopeManager().active();
    if (scope != null) {
        Span span = scope.span();
        Tags.ERROR.set(span, true);
        span.log(Map.of(Fields.EVENT, "error", Fields.ERROR_OBJECT, e, Fields.MESSAGE, e.getMessage()));

    }
    LOG.error(message, e);
}

However, so far I was not able to find opentracing configuration options that would allow to add the application logs to the tracer automatically by default. Basically, it seems that it is expected that dev would add extra logs to tracer programmatically if needed. Also, after investigating tracing more it appeared to be that normally logging and tracing are handled separately and adding all the application logs to the tracer is not a good idea (tracer should mainly include sample data and tags for request identification)

Solution

  • https://github.com/opentracing-contrib/java-spring-cloud project automatically sends standard logging to the active span. Just add the following dependency to your pom.xml

    <dependency>
   <groupId>io.opentracing.contrib</groupId>
   <artifactId>opentracing-spring-cloud-starter</artifactId>
</dependency>

    Or use this https://github.com/opentracing-contrib/java-spring-cloud/tree/master/instrument-starters/opentracing-spring-cloud-core starter if you want only logging integration.