AWS X-Ray trace segments are missing or not connected

Question

I have code that creates a segment when a queue is being read. In the first function (within the same lambda) I have this:

import * as AWSXRay from 'aws-xray-sdk'; // (using TypeScrpt)
AWSXRay.enableManualMode();
var segment1 = new AWSXRay.Segment("A");

In the second function (within the same lambda), called from the first, I have something like this:

var segment2 = new AWSXRay.Segment("B", segment1.trace_id, segment1.id);

Instead of seeing

*->A->B

On the AWS graph (on the website), I see:

*->A
*->B

...where they are not even associated, even though they have the same tracing ID, and the parent IDs are properly set. I seem to be missing something but not sure what...?

I even tried to pull X-Amzn-Trace-Id from the API request to use that as the root tracking ID for everything but that didn't work either.

This is the JSON for the first segment (A):

{
  "Duration": 0.808,
  "Id": "1-5d781a08-d41b49e35c3c0f38cdbd4912",
  "Segments": [
    {
      "Document": {
        "id": "74c99567f73185ce",
        "name": "router",
        "start_time": 1568152071.979,
        "end_time": 1568152072.787,
        "parent_id": "ef34fc0bcf23bbbe",
        "aws": {
          "xray": {
            "sdk": "X-Ray for Node.js",
            "sdk_version": "2.3.6",
            "package": "aws-xray-sdk"
          }
        },
        "service": {
          "version": "unknown",
          "runtime": "node",
          "runtime_version": "v10.16.3",
          "name": "unknown"
        },
        "trace_id": "1-5d781a08-d41b49e35c3c0f38cdbd4912"
      },
      "Id": "74c99567f73185ce"
    }
  ]
}

This is the JSON for the second segment (B):

{
    "Duration": 0.801,
    "Id": "1-5d781a08-d9626abbab1cfbbfe4ff0dff",
    "Segments": [
        {
            "Document": {
                "id": "e2b4faaa6538bbb2",
                "name": "handleCreateLoad",
                "start_time": 1568152071.98,
                "end_time": 1568152072.781,
                "parent_id": "74c99567f73185ce",
                "aws": {
                    "xray": {
                        "sdk": "X-Ray for Node.js",
                        "sdk_version": "2.3.6",
                        "package": "aws-xray-sdk"
                    }
                },
                "service": {
                    "version": "unknown",
                    "runtime": "node",
                    "runtime_version": "v10.16.3",
                    "name": "unknown"
                },
                "trace_id": "1-5d781a08-d9626abbab1cfbbfe4ff0dff",
                "subsegments": [
                  {
                    "id": "08ccf2f374364066",
                    "name": "...-CreateLoad",
                    "start_time": 1568152071.981,
                    "end_time": 1568152072.781
                  }
                ]
            },
            "Id": "e2b4faaa6538bbb2"
        }
    ]
}

It's quite clear the the parent ID for 'B' (74c99567f73185ce) points to "A"'s ID, but the graph does not connect them.

Also, I think _x_amzn_trace_id should be set when the lambda executes, but it is not. That may be root of my issues.

Answer 1

Turns out process.env._x_amzn_trace_id, required by the AWS XRay SDK, does NOT exist until the handler is called. It may help others to know what I went through:

1. At first I tried to get the trace details for the current lambda on start up (before the handler is called) to connect my new segments, but it didn't work. I have many handlers in the same project, so getting the lambda segment on startup is what I was hoping to do.
2. I then proceeded to create a main lambda segment (thinking I had to create the first segment myself) but all it did was create an orphaned segment. To make matters worse, each segment creates a new trace ID if one is not provided, and since I could not get the trace ID from the global start-up scope, nothing was connecting. The proper trace ID is important to pass along from start to finish for each request to make sure the calls down-stream are tracked properly.
3. Dumping of the environment variables before the handler is called and after clearly showed the trace ID is not provided until just before the handler gets called. It's sad that most of the online examples don't even bother to warn about this. I then moved the called to AWSXRay.getSegment() at the start of the lambda handler, then passed the details onto the child segments.
4. DO NOT set context.callbackWaitsForEmptyEventLoop = false while also calling the callback(error, response) callback passed to the lambda handler. Doing so will terminate the lambda without waiting for segment update events to flush to the daemon, resulting in orphaned segments. :(

Note: This documentation is lacking: https://docs.aws.amazon.com/xray-sdk-for-nodejs/latest/reference/ It states "You can retrieve the current segment or subsegment at any time" when in fact there are some times when you cannot. It's too bad there are no proper examples using actual working NodeJS Lambda code, instead of isolated lines of code thrown everywhere.