How to pattern multiline XML using RegEx in NXLog

Question

I am trying to parse a custom log file into JSON using the nxLog parser to_json() so that I can then ship them into my ElasticSearch instance. I am going to be splitting these into three separate fields, date, log type indicator, and message.

Following is the format of these logs.

9/10/2015 11:30:05 AM [0-1-1-Pos.xaml.cs-1607] Post button clicked

9/10/2015 11:30:17 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION>
  <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE>
  <COMMAND>STATUS</COMMAND>
  <MAC_LABEL>XX</MAC_LABEL>
  <MAC>xOel7QeyKoXaddiyrEeWKRI1DlF9sHzUNfZHFI/gAko=</MAC>
 <COUNTER>XXX</COUNTER>
</TRANSACTION>

9/10/2015 11:30:17 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE>
  <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
  <RESULT>OK</RESULT>
  <RESULT_CODE>-1</RESULT_CODE>
  <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
  <COUNTER>221</COUNTER>
  <SECONDARY_DATA>12</SECONDARY_DATA>
  <MACLABEL_IN_SESSION>P_061</MACLABEL_IN_SESSION>
  <SESSION_DURATION>00:00:16</SESSION_DURATION>
  <INVOICE_SESSION>XX</INVOICE_SESSION>
  <SERIAL_NUMBER>XX</SERIAL_NUMBER>
</RESPONSE>`

I've been able to parse date stamp and the error selector (everything within brackets) using PERL regex syntax as follows.

1. ^(\d\d|\d)/(\d\d|\d)/(\d\d\d\d)\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM) 
2. \[(.*)\] 

1. Date
2. Log Type Identifier
3. Message which will be what I am trying to figure out.

But I cannot figure out how to pull everything between the selector and the new line. So in this instance I'd like my message to be the XML code until the newline. Does anyone have advice on how I can retrieve the data?

Answer 1

You should be able to use nxlog's xm_multiline module and specify the regexp in the HeaderLine directive. If you add a capturing rule to the regexp to match the XML part (stuff after the [..]) then you should be able to parse the XML with xm_xml's parse_xml().

There is a similar example here.