Link spans running in different processes with OpenTelemetry

Question

I am just starting with OpenTelemetry and have created two (micro)services for this purpose: Standard and GeoMap.

The end-user sends requests to the Standard service, who in turn sends requests to GeoMap to fetch informations before returning the result to the end-user. I am using gRPC for all communications.

I have instrumented my functions as such:

For Standard:

func (s *server) GetStandard(ctx context.Context, in *pb.GetStandardRequest) (*pb.GetStandardResponse, error) {

    newCtx, span := otel.Tracer(name).Start(ctx, "GetStandard")
    span.SpanContext()
    defer span.End()

    countryInfo, err := geoMapServiceClient.GetCountry(newCtx,
        &pb.GetCountryRequest{
            Name: in.Name,
        })

    ....
    
    return &pb.GetStandardResponse{
        Standard: standard,
    }, nil

}

For GeoMap

func (s *geomapService) GetCountry(ctx context.Context, in *pb.GetCountryRequest) (*pb.GetCountryResponse, error) {

    _, span := otel.Tracer(name).Start(ctx, "GetCountry")
    defer span.End()

    span.SetAttributes(attribute.String("country", in.Name))

    ...

    return &pb.GetCountryResponse{
        Country: &country,
    }, nil

}

Both services are configured to send their traces to a Jaeger Backend:

func tracerProvider(url string) (*tracesdk.TracerProvider, error) {
    // Create the Jaeger exporter
    exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(url)))
    if err != nil {
        return nil, err
    }
    tp := tracesdk.NewTracerProvider(
        tracesdk.WithBatcher(exp),
        tracesdk.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(service),
            attribute.String("environment", environment),
            attribute.Int64("ID", id),
        )),
    )
    return tp, nil
}

func main() {

    tp, err := tracerProvider("http://localhost:14268/api/traces")
    if err != nil {
        log.Fatal(err)
    }

    defer func() {
        if err := tp.Shutdown(context.Background()); err != nil {
            log.Fatal(err)
        }
    }()

    otel.SetTracerProvider(tp)
    
    ...
}

I would expect the span generated in the GeoMap service to be a child span of the Standard service (since it is the Standard service who is making the request through the geoMapServiceClient). However these spans are totally unrelated in Jaeger:

What am I missing here?

EDIT:

I have managed to include interceptors as @Peter suggested in the comments. Now spans are linked to each other and I can see that my parent span from the StandardService is making a request to the GeoMapService:

However, I have defined an attribute in my function call (see the span.SetAttributes(attribute.String("country", in.Name))) but I am unable to see that value from within my child span.

For that I have to switch to the other trace generated by the service GeoMap:

I don't need to have two different traces actually, I would like to have everything in the same trace (one end-user query should result in a single trace). How should I change my code from here?

Answer 1

Your child span has no way of knowing where its parent comes from. For that to happen, you need your service to inject the proper HTTP headers into your gRPC requests, and to read it back afterwards.

OpenTelemetry provides a propagators API, which allows you to do that.

While you can do that manually within your code to inject/extract the headers, doing so is not the recommended way. If you add the otelgrpc interceptors to your applications, that propagation will be done automatically, and your spans will become childs of each other.