Looking for keywords within a grok %{QS:message}

Question

I've searched up and down and was wondering or if this is even an option that is possible within Grok. So my log files are filtered just fine. Except, the %{QS:message} is what contains my ERROR, WARNING, POST, GET etc. I want to be able to query against those in Kibana but do not have them as an option. Is there anything I can do to make these keywords available that are coming back from logstash to kibana

Answer 1

its probably easiest if you analyze this step by step. add the following to your output section stdout { codec => rubydebug }

When processing a log message it details you all the fields and their values that were created by logstash during processing of your logmessage. As long as you did not specify anything fancy in your elasticsearch output these fields will be stored in elasticsearch (along with their unanalysed .raw counterparts). If that looks okay you could then take a look at the elastisearch side of things using its rest interface interactively (using curl) or by installing the elasticsearch kopf plugin (or something similar) to check out whats actually been stored in elasticsearch.

EDIT: Regarding your last comment.

Assuming your log data looks like this "POST: Form submitted from shoppingcart.php"

You could either use an if statement with a regexp to simply check if your message contains a given character sequence (like POST) and then use mutate filter to add a field or a tag to your event like so:

if [logmessage] =~ /POST/ {
    mutate { add_tag => ["POST"] }
    # or you could also do something like:
    mutate { add_field => ["method","POST"]}
}

Instead of the if statement you could also use a grok filter to further parse your message like so:

grok {
      match => ["logmessage", "(?<method>POST)"]
      tag_on_failure => []
}

Depending on the structure of your log entries and the complexity of what you want to extract its prefereably use method a. or b. If your logs are very structured you could simply build one grok filter that takes care of all variants. Let's say if all your lines looks like this:

method|returnCode|messageText

you could do a simple grok filter like

%{DATA:method}|%{DATA:returnCode}|{GREEDYDATA:messageText}

that takes care of all your lines and parses your logs into fields for kibana analysis.

If however your logs are very unstructured and you only want to look up a small number of keywords you can also go with the if regexp flavour...