Tracing requests over their lifetime … through Docker Containers?

Question

Tracing makes finding parts in code, worthwhile a developers time and attention, much easier. For that reason, I attached Jaeger as tracer to a set of microservices inside Docker containers. I use Traefik as ingress controller/ service-mesh to route and proxy requests.

The problem I am facing is, that something's wrong with the tracing config in Traefik. Jaeger can not find the span context to connect the single/ service-dependend spans to a whole trace.

The following line appears in the logs:

{
  "level":"debug",
  "middlewareName":"tracing",
  "middlewareType":"TracingEntryPoint",
  "msg":"Failed to extract the context: opentracing: SpanContext not found in Extract carrier",
  "time":"2021-02-02T23:16:51+01:00"
}

What I tried/ searched/ confirmed so far:

• I already checked ports (they are open inside the Docker host network) and everything's reachable. So interconnectivity is not the problem here.
• The forwarding of headers is set via Docker Compose labels: loadbalancer.passhostheader=true.

---

The following snippets describe the Docker Compose setup.

Traefik: Ingress Controller

This is a stripped down version of the traefik Container.

# Network
ROOT_DOMAIN=example.test
DEFAULT_NETWORK=traefik

---
version: '3'
services:
    image: "traefik:2.4.2"
    hostname: "controller"
    restart: on-failure
    security_opt:
      - no-new-privileges:true
    ports:
      - "443:443"
      - "80:80"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
      - "8082:8082"
      - "8083:8083"
    networks:
      - default
    working_dir: /etc/traefik
    volumes:
      - /private/etc/localtime:/etc/localtime:ro
      - ${PWD}/controller/static.yml:/etc/traefik/traefik.yml:ro
      - ${PWD}/controller/dynamic.yml:/etc/traefik/dynamic.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - cert-storage:/usr/local/share/ca-certificates:ro
      - ${PWD}/logs/traefik:/var/log/traefik

volumes:
  cert-storage:
    driver_opts:
      type: none
      o: bind
      device: ${PWD}/certs/certs

networks:
  default:
    external: true
    name: ${DEFAULT_NETWORK}

Traefik is set up using the file provider as base and Docker Compose labels on top of it:

# static.yaml (Traefik conf)
debug: true

log:
  level: DEBUG
  filePath: /var/log/traefik/error.log
  format: json

serversTransport:
  insecureSkipVerify: true

api:
  dashboard: true
  insecure: true
  debug: true

providers:
  docker:
    exposedByDefault: false
    swarmMode: false
    watch: true
    defaultRule: "Host(`{{ normalize .Name }}.example.test`)"
    endpoint: "unix:///var/run/docker.sock"
    network: traefik
  file:
    filename: /etc/traefik/dynamic.yml
    watch: true

tracing:
  serviceName: "controller"
  spanNameLimit: 250
  jaeger:
    samplingType: const
    samplingParam: 1.0
    samplingServerURL: http://tracer:5778/sampling
    localAgentHostPort: 127.0.0.1:6831
    gen128Bit: true
    propagation: jaeger
    traceContextHeaderName: "traefik-trace-id"
    collector:
      endpoint: http://tracer:14268/api/traces?format=jaeger.thrift

Jaeger: Open Tracing/ Open Telemetry

---
version: '3'
services:
  tracer:
    image: "jaegertracing/all-in-one:1.21.0"
    hostname: "tracer"
    command:
      - "--log-level=info"
      - "--admin.http.host-port=:14269"
      - "--query.ui-config=/usr/local/share/jaeger/ui/conf.json"
    environment:
      SPAN_STORAGE_TYPE: memory
    restart: on-failure
    security_opt:
      - no-new-privileges:true
    expose:
      - 5775/udp
      - 6831/udp
      - 6832/udp
      - 5778
      - 14250
      - 14268
      - 14269
      - 14271
      - 16686
      - 16687
    volumes:
      - /private/etc/localtime:/etc/localtime:ro
      - ${PWD}/tracer/conf:/usr/local/share/jaeger
      - ${PWD}/logs/jaeger:/var/log/@TODO
      - cert-storage:/usr/local/share/ca-certificates
    networks:
      - default
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=${DEFAULT_NETWORK}"
      # Admin UI router
      - "traefik.http.routers.tracer-router.rule=Host(`tracer.$ROOT_DOMAIN`)"
      - "traefik.http.routers.tracer-router.entrypoints=https"
      - "traefik.http.routers.tracer-router.tls=true"
      - "traefik.http.routers.tracer-router.tls.options=default"
      - "traefik.http.routers.tracer-router.service=tracer"
      # Service/ Load Balancer
      - "traefik.http.services.tracer.loadbalancer.passhostheader=true"
      - "traefik.http.services.tracer.loadbalancer.server.port=16686"
      - "traefik.http.services.tracer.loadbalancer.server.scheme=http"

Answer 1

The problem actually was with the traceContextHeaderName. Sadly I can not tell exactly what the problem was as the git diff only shows that nothing changed around traefik and jaeger at the point where I fixed it. I assume config got "stuck" somehow. I tracked down the related lines in source, but as I am no Go-Dev, I can only guess if there's a bug.

What I did was to switch back to uber-trace-id, which magically "fixed" it. After I ran some traces and connected another service (node, npm jaeger-client with process.env.TRACER_STATE_HEADER_NAME set to an equal value), I switched back to traefik-trace-id and things worked.