We are sending CF logs message to an ELK stack. Multiline logs message are broken out into several log messages in Logstash. One end per line of the multiline log message. This is problematic when stack traces dumped to the log. Each line of the stack trace is translated into a log message. Trying to view this through Kibana is nearly impossible. Logstash provides a Grok feature allowing for the manipulation of the log messages. One common solution is to create a Grok filter that using a timestamp to indicate when a log entry starts and to combine all lines until the next timestamp into one log message. The problem is that CF adds a timestamp to every line. Has anyone come up with a good Grok expression to handle multiline log messages coming out of CF?
Here's a snippet of a log from Cloud Foundry. In Logstash, each line is seen as a separate log message
2015-07-24T10:37:59.93-0600 [App/0] OUT org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://11.11.11.11:9191/someendpoint":Connect to 11.11.11.11:9191 [/11.11.11.11] failed: Connection refused; nested exception is org.apache.http.conn.HttpHostConnectException: Connect to 11.11.11.11:9191 [/11.11.11.11] failed: Connection refused
2015-07-24T10:37:59.93-0600 [App/0] OUT at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:580)
2015-07-24T10:37:59.93-0600 [App/0] OUT at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:530)
2015-07-24T10:37:59.93-0600 [App/0] OUT at org.springframework.web.client.RestTemplate.postForLocation(RestTemplate.java:305)
2015-07-24T10:37:59.93-0600 [App/0] OUT at com.digitalglobe.p2020.eventservice.sendevent.EventSenderCommand.run(EventSenderCommand.java:104)
From the logs you provided I see that you have the first message of the multiline and then the rest of the messages are indented by additional two spaces after the OUT part.
This means that you should be able to catch the first line using the following pattern
^%{TIMESTAMP_ISO8601}\s+\[%{DATA}]\s+\S+\s{2}\S
Please notice that it should not catch the preceding lines and only the first one and if you change the value of s{2} to s{4} you will catch the presiding lines and not the first one (the opposite of what you want).