Search code examples
logstashlogstash-grok

Log parsing with dissect

input is the following message text:

{"timestamp":"2018-08-02 15:56:46,569","level":"DEBUG","class":"classname","method":"run","line":"730","thread":"threadname","message":"messagetext"}

This is my filter part of logstash config:

filter {
  if "pattern_09" in [tags] {
    mutate {
      gsub => [ "message" , "{" , "",
                "message" , "}" , "" ] }
    dissect {
      mapping => { "message" => "%{Timestamp},%{+Timestamp/2},%{level},%{class},%{method},%{line},%{thread},%{mymessage}"}
    }
    mutate {
      gsub => [ "Timestamp" , "timestamp", "",
                "Timestamp" , "\"", "" ,
                "level" , "level" , "" ,
                "level" , ":", "" ,
                "level" , "\"", "" ,
                "class" , "class" , "" ,
                "class" , ":", "" ,
                "class" , "\"", "" ,
                "method" , "method" , "" ,
                "method" , ":", "" ,
                "method" , "\"", "" ,
                "line" , "line" , "" ,
                "line" , ":", "" ,
                "line" , "\"", "" ,
                "thread" , "thread" , "" ,
                "thread" , ":", "" ,
                "thread" , "\"", "" ,
                "mymessage" , "mymessage" , "" ,
                "mymessage" , ":", "",
                "mymessage" , "\"", "" ]
    }
    date {
      match => [ "Timestamp", "yyyy-dd-MM HH:mm:ss,SSS" ]
    }
  }
}

In then my timestamp looks like this:

:2018-08-02 15:56:46,569

and i am not able to remove/erase the first occurence of ":". I have tried it with 

date {
  match => [ "Timestamp", "yyyy-dd-MM HH:mm:ss,SSS" ]
}

but that has no effect.

Solution

  • Why not use the grok filter? It's much more convenient for such pattern matching.

    For the above example, it can be done as below

    grok {
  match => ["message" ,'%{GREEDYDATA}"timestamp":"%{DATA:Timestamp}"%{GREEDYDATA}"level":"%{DATA:Level}"%{GREEDYDATA}"class":"%{DATA:Class}"%{GREEDYDATA}"method":"%{DATA:Method}"%{GREEDYDATA}"line":"%{DATA:Line}"%{GREEDYDATA}"thread":"%{DATA:Thread}"%{GREEDYDATA}"message":"%{DATA:Message}"%{GREEDYDATA}']
}

date {
  match => [ "Timestamp", "yyyy-dd-MM HH:mm:ss,SSS" ]
}

    DATA and GREEDYDATA are just patterns and there are more we can use here: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns

    More about the grok filter here: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html and you can test your grok patterns here:https://grokdebug.herokuapp.com/