AWS streaming multi-line log files from CloudWatch to ELK

Question

We are streaming app logs from CloudWatch to AWS ELK. Our microservices are written in Java and so I am only concentrating on those. A typical java exception stack trace when logged looks like this:

Exception in thread "main" java.lang.NullPointerException
    at com.example.myproject.Book.getTitle(Book.java:16)
    at com.example.myproject.Author.getBookTitles(Author.java:25)
    at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Normally, this will be ingested line by line in ELK stack, which breaks the entire message.

Usually, For entire stack trace to be ingested as a single message, one can configure multiline plugin either in Logstash or Filebeat.

Any idea how to enable multiline while streaming log files from CloudWatch to ELK by AWS lambda?

Answer 1

It turned out to be a know issue with AWS cloudwatch logger, see RichardBronosky's comment at https://github.com/visionmedia/debug/issues/296

I did some testing and found that CloudWatch log entries can be made multiline by using \r as the line delimiter. Using either \n (Unix) or \r\n (DOS) line endings will result in separate entries

So I fixed it by adding following ExceptionHandler to Spring Boot RestController

@ExceptionHandler(Throwable.class)
void handleUnhandledExceptions(Throwable e, HttpServletResponse response) throws IOException {
    StringWriter buffer = new StringWriter();
    e.printStackTrace(new PrintWriter(buffer));
    log.error(buffer.toString().replace("\n", "\r"));
    response.sendError(HttpStatus.INTERNAL_SERVER_ERROR.value(), e.getMessage());
}