how to parse and extract specific fields and store it into another field in logstash filter?

Question

I have application log files which are shipped to the logstash using Filebeat. the log file will have content something like this, with module name and the time taken to process in it

[10/08/2020#11:25:45:451] #SVS#SVS#NA#NA#NA#-#DE#00000199#DE_ONL_DC_SERVER1_NODE05_PRD##[8/10/2020#11:25:45:451] #O#-#-#-#Module1#1#-#5#-#-#-#CoreB#-

I want to extract only 3 fields from the above log, the modulename {Module1} , Time taken to process{5}, Server Name{DE_ONL_DC_SERVER1_NODE05_PRD} , so that I can plot them into a dashboard of those terms in kibana.

I am quite new to the elk stack and still exploring on the various filter options in logfilter to achieve the above, Any help will be appreciated. Thanks

Answer 1

Yes, exactly, if all your log lines look exactly the same, I would leverage the dissect filter. A pattern like this one should do:

filter {
  dissect {
    mapping => {
      "message" => "[%{@timestamp}] #%{?ignored}#%{?ignored}#%{?ignored}#%{?ignored}#%{?ignored}#-#%{?ignored}#%{?ignored}#%{serverName}##[%{?ignored}] #%{?ignored}#%{?ignored}#%{?ignored}#%{?ignored}#%{moduleName}#%{?ignored}#%{?ignored}#%{processTime}#%{?ignored}#%{?ignored}#%{?ignored}#%{?ignored}#%{?ignored}
"
    }
  }
}

This would extract the following fields into the event:

{
    "@timestamp": "10/08/2020#11:25:45:451",
    "message": "[10/08/2020#11...",
    "serverName": "DE_ONL_DC_SERVER1_NODE05_PRD",
    "moduleName": "Module1",
    "processTime": 5
}