Not seeing manually created nested AWS XRay subsegments in AWS console

Question

I'm trying to instrument a web server written in Go that provides REST API, that runs in a container on AWS ECS. I'm running the server in Debug in VSCode for now, working on a proof of concept that would show traces for the different endpoints, with major functions as subsegments. I've instrumented the router in our middleware like this:

h = xray.Handler(xray.NewFixedSegmentNamer("myappname"), h)

And function calls made from the various handler functions are instrumented by passing the request context in, then having this:

_, subSeg := xray.BeginSubsegment(ctx, "get-user")

calculateUsefulInfo(ctx)

subSeg.Close(nil)

Then that calculateUsefulInfo() function can call other functions, passing along the context (ctx) and internally doing the same thing (another BeginSubsegment+subSeg.Close) with a different subsegment name.

I have the AWS XRay daemon running, with appropriate permissions, and I see the traces appear in the AWS console. However, I only see one level of nesting.

I turned on 100% sampling in AWS. I'm running the XRay daemon in local mode, dev-level logging. Any idea what I'm missing here?

Answer 1

The parent segment is stored in the context, so if you are only passing in the top level context to the other functions, it will only generate segments for that parent.

To achieve your desired nesting level you have to use the context provided by xray.BeginSubsegment. This context contains a new segment that can be traced back to the parent.

Example with nested segments:

package main

import (
    "net"
    "net/http"
    "time"

    "github.com/aws/aws-xray-sdk-go/xray"
    "github.com/davecgh/go-spew/spew"
)

type ConsoleEmitter struct {
}

func (c *ConsoleEmitter) Emit(seg *xray.Segment) {
    spew.Dump(seg)
}

func (c *ConsoleEmitter) RefreshEmitterWithAddress(raddr *net.UDPAddr) {
    return
}

var _ xray.Emitter = (*ConsoleEmitter)(nil)

func init() {
    xray.Configure(xray.Config{
        DaemonAddr:     "127.0.0.1:2000", // default
        ServiceVersion: "1.2.3",
        // console emitter to view the hierarchy of the traces locally
        // without the xray daemon
        Emitter: &ConsoleEmitter{},
    })
}

func main() {
    http.Handle("/", xray.Handler(xray.NewFixedSegmentNamer("myApp"), http.HandlerFunc(top)))
    http.ListenAndServe(":7000", nil)
}

func top(w http.ResponseWriter, r *http.Request) {
    // use the context provided by xray for nested hierarchy
    ctx, subSeg := xray.BeginSubsegment(r.Context(), "top")
    _, childSeg := xray.BeginSubsegment(ctx, "top-sleep")
    time.Sleep(time.Millisecond * 50)
    childSeg.Close(nil)
    middle(w, r)
    subSeg.Close(nil)
}

func middle(w http.ResponseWriter, r *http.Request) {
    ctx, subSeg := xray.BeginSubsegment(r.Context(), "middle")
    _, childSeg := xray.BeginSubsegment(ctx, "middle-sleep")
    time.Sleep(time.Millisecond * 100)
    childSeg.Close(nil)
    bottom(w, r)
    subSeg.Close(nil)
}

func bottom(w http.ResponseWriter, r *http.Request) {
    _, subSeg := xray.BeginSubsegment(r.Context(), "bottom")
    w.Write([]byte("Hello!"))
    subSeg.Close(nil)
}