Search code examples
logstashlogstash-groklogstash-configuration

logstash filters to parse comma separated log file

I have a log file which looks like this:

2019-04-19 08:30:06,477 DEBUG [org.mobicents.smsc.library.CdrGenerator] 2019-04-19 08:30:01.960,466,2,1,01706488575,1,1,success,SMPP,message,3rdPartyName,10614279,null,null,01702993001,412012102179043,null,null,0170260020,0,0,null,0,0,,,,2393,"*466#
nxxxxxxxxxxxxxxx","",,,

I want to filter this logs on logstash and only get and store few columns. as you see the logs column are comma separated, I want to ignore all columns except:

  • the date and time part from column 2 ( i want to get date and time as 2 different columns)
  • column 9 and column 12
Solution

  • Logstash has a CSV filter that would give you every column in a field. The upside is that it would be parsed correctly. The downside is that you'd then have to remove the columns you don't want from the event.

    You can also write your own pattern for the grok filter. There are a lot of examples and tutorials on the web about this.