Grabbing multiple lines using Grok

The following is a log sample I need to parse using logstash and the logstash grok filter:

2018-02-12 15:17:39.216 [DEBUG] [    60] [CashTransactionReportCommand] [4564 456] - Xml of valid cash: <NewDataSet>
  <Table>
    <transaction_id>546464</transaction_id>
    <device_trans_id>24</device_trans_id>
    <value>3.5000</value>
    <product_code>40</product_code>
    <product_pa_code>E1</product_pa_code>
    <catalog_number />
    <decimal_place>2</decimal_place>
    <site_id>2</site_id>
    <machineSeTime>2018-02-12T17:17:39.273+00:00</machineSeTime>
    <payment_method_id>3</payment_method_id>
    <actor_id>4566</actor_id>
    <operator_id>55</operator_id>
  </Table>
</NewDataSet>

I almost have everything I need:

%{TIMESTAMP_ISO8601:log_timestamp} \[%{LOGLEVEL:loglevel}\] \[%{DATA:snId}\] \[%{WORD:snName}\] (?<test>\[\d+ \d+\]) %{GREEDYDATA:logmessage}

My only problem with the "logmessage". I need it to contain everything passed "[4564 456]" until the end of the example.

Solution

In order to be able to parse the message, including the XML, you'll have to group all the lines in the same logstash event, so that when using the grok filter, the message field contains the whole message. This can be done:

in logstash with the multiline codec

Multiline in logstash

Multiline codec documentation

in filebeat with the multiline option

Multiline in filebeat

Documentation of multiline option in filebeat configuration