Search code examples
regexparsingloggingclouderastreamsets

How to parse multiple lines record of Log file using StreamSets?


I'm using StreamSets to parse a Log file, the problem that StreamSets parses line by line and my log record is multiple lines, something like this

00:01:03.930 [WebContainer : 41] Outbound message:
00:01:03.930 [WebContainer : 41] Values to hide NewPassword -- mask -- .+

I tried regex and grok patterns, but the new line tag doesn't work for me. So, how to make StreamSets parses the record as multiple lines?


Solution

  • I created custom processor to parse my file. I followed this tutorial and worked just fine and like i wanted https://github.com/streamsets/tutorials/tree/master/tutorial-origin