Handling exception stack trace while writing serial no in json logs using Json template layout log4j2

Question

I am using json template layout for writing json logs for my ecs service to s3 via kinesis firehose and firelens.

Following is the configuration for json template layout I am using -

{
   "context_map": {
       "$resolver": "mdc"
   },
   "serial_no": {
     "$resolver": "pattern",
     "pattern": "%sn"
   },
   "timestamp": {
     "$resolver": "timestamp",
     "pattern": {
       "format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
       "timeZone": "UTC"
     }
   },
   "level": {
     "$resolver": "level",
     "field": "name"
   },
   "logger_name": {
     "$resolver": "logger",
     "field": "name"
   },
   "message": {
     "$resolver": "message",
     "stringified": true
   },
   "thread": {
     "$resolver": "thread",
     "field": "name"
   },
   "exception": {
     "exception_class": {
       "$resolver": "exception",
       "field": "className"
     },
     "exception_message": {
       "$resolver": "exception",
       "field": "message",
       "stringified": true
     },
     "stacktrace": {
       "$resolver": "exception",
       "field": "stackTrace",
       "stringified": true
     }
   },
   "class": {
     "$resolver": "source",
     "field": "className"
   }
 }

Following is snapshot of log configuration -

     Console:
       - Name: Console
         Target: SYSTEM_OUT
         JsonTemplateLayout:
           eventTemplateUri: "file:/opt/amazon/log-configuration/JsonLayoutTemplate.json"
           prettyPrintEnabled: true
           stackTraceEnabled: true

Following is the sample log generated in case of exception -

{
    "container_id": "1f1319d278264d3dwewdeddewdewdewdew-849099158",
    "container_name": "test-container",
    "context_map": {
        "RequestId": "28289758-ab3b-4f8c-b3f1-freferfc"
    },
    "exception": {
        "exception_class ": "com.xyz.exception”,
        "exception_message": "Problem processing the SQS message with body helllo world, exception: Error while deserializing the message from test - message - queue.",
        "stacktrace": "com.xyz.exception: Problem processing the SQS message with body helllo world, exception: Error while deserializing the message from test - message - queue.\n\ .... stackTrace”
    },
    "level": "ERROR”,
    "logger_name": "com.xyz.handler",
    "message": "Seems to be impossible to process message: helllo world",
    "serial_no": "52 com.xyz.handler: Problem processing the SQS message with body helllo world, exception: Error while deserializing the message from test - message - queue.\n\ .... stackTrace",
    "source": "stdout",
    "thread": "test-thread",
    "timestamp": "2021-04-06 T14:30:02.122 Z"
}

As per the documentation Pattern Resolver delegates to PatternLayout and pattern layout provides %sn for getting sequence number.

We do need sequence number, since we can get a lot of logs simulatenously , so in order to differentiate between logs, we will be relying on sequence number, requestId params in those json logs while querying logs in AWS Athena.

Also, I have seen few log instances where the serial no is null (not the exception logs)

How can I prevent serial_no from including the exception stacktrace within it ? And ensure that I always get serial_no field in the logs ?

Answer 1

When you delegate from JsonTemplateLayout to PatternLayout via pattern resolver, the LogEvent is passed along as is. Once PatternLayout finds out that there is an exception attached to the LogEvent, it appends the stack trace to the emitted output, because the created PatternLayout instance inherited the stackTraceEnabled=true flag which you set for JsonTemplateLayout. Hence, you need to disable the stack traces for PatternLayout as follows:

{
  "$resolver": "pattern",
  "pattern": "%sn",
  "stackTraceEnabled": false
}

Note that this will still produce sequence numbers of type string. If you want to produce sequence numbers of type number, you can use timestamp resolver in the following fashion:

{
  "$resolver": "timestamp",
  "epoch": {
    "unit": "nanos",
    "rounded": true
  }
}

In the meantime, I have created LOG4J2-3067 to add a sequence number resolver to JsonTemplateLayout.