Search code examples
amazon-web-servicesaws-xrayopen-telemetry

AWS X-Ray not initializing a new segment

I'm using OpenTelemetry with AWS X-Ray to trace E2E messaging between:

Producer (JVM) -> Kafka Broker -> Consumers (Multiple, Python-based)

Generated traces are then send to AWS OTEL Collector which forwards them to AWS X-Ray.

However when I see them from X-Ray consumer trace is displayed as a sub-segment of producer:

enter image description here

I'm expecting to see consumer as a separate segment.

I've also tried to use AWS X-Ray SDK in the consumer side to explicitly initialize a new segment as following:

val traceId = TraceIdData(String(record.headers().headers("X-Amzn-Trace-Id").first().value()))
logger.debug("Trace Id {}, Parent Id {}", traceId.root, traceId.parent)
AWSXRay.beginSegment("Simple Kafka Consumer", traceId.root, traceId.parent)
... some processing ...
AWSXRay.endSegment()

However this is still shown as a sub-segment in AWS X-Ray even though I'm beginning a new segment and specifying: trace_id and parent_id. I have confirmed that both attributes are not null and valid.

How can I generate a new segment in AWS X-Ray for this ongoing trace? Preferably, I'd like to do this using OpenTelemetry (agent/sdk/collector), but using AWS X-Ray SDK is also a possibility.

Solution

  • I had a chat with AWS X-Ray team and they helped me solve my issue.

    The reason for this issue is that awsxray exporter only creates new segments for Spans when kind=SpanKind.SERVER.

    I have implemented this (in Python) using OpenTelemetry SDK and explicitely declaring kind=SpanKind.SERVER when creating a new span

    processing_span = self.tracer.start_span(self.service_name, context=current_context, kind=SpanKind.SERVER)

    Which gives me my desired result of seeing multiple consumers in the Service Map Service Map

    Unfortunately OpenTelemetry java instrumentation for Kafka on consumer side is automatically creating a new span where kind=CONSUMER which cannot be modified (as part of automatic instrumentation) so the only way to get my desired result is through manual instrumentation.