Analyze sensu log with logstash

Question

Here is my sensu log in the server:

{"timestamp":"2015-01-21T09:43:21.387501+0700","level":"info","message":"publishing check result","payload":{"client":"local.com","check":{"name":"instance_xxx_check","issued":1421808200,"command":"xargs -I{} sh -c '/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/check-http.rb -u {}' < /etc/sensu/conf.d/live/list/xxx.txt","handlers":["default","mailer"],"interval":60,"subscribers":["live"],"executed":1421808200,"duration":1.317,"output":"CheckHTTP OK: 200, http://link1.com\nCheckHTTP CRITICAL: Request error: http://link2.com\nCheckHTTP OK: 200, http://link3.com\n","status":123}}}

this is in json format, you can use json parse to view it.

And after pass logstash filter, it will parse in fields like this image:

https://i.sstatic.net/4KA0i.jpg

And now I want to add a field named "error" which just contain information of critical "http://link.com". It's mean, if filter match CheckHTTP CRITICAL in the "payload.check.output" field, it'll add the error link to new "error" field

And this is my config in logstash filter:

if [type] == "sensu" {
            grok {
                match => [ "payload.check.output", "%{CISCO_REASON}: Request error: %{URI}" ]
            }
            mutate {
                add_field => { "error" => "%{payload.check.output}" }
                remove_field => [ "timestamp" ]
            }
        }

but there is nothing happen

Answer 1

You can try to use this config to access the nested json message.

input {
    stdin{}
}

filter {
    json {
            source => "message"
    }
    if [payload][check][output] =~ /CheckHTTP CRITICAL/
    {
        ruby
        {
            code => "
                errormsg = '';
                msg = event['payload']['check']['output'];
                splitmsg = msg.split(/\n/)
                for splitlog in splitmsg
                     if splitlog.include? 'CheckHTTP CRITICAL'
                         errormsg = errormsg + splitlog.split('error: ')[1];
                     end
                end
                event['error'] = errormsg
            "
        }
    }
}


output {
    stdout {
            codec => rubydebug
    }
}

The above config can parse out the error link and insert to error field.

If you found that your requirement is hard to implemented by logstash plugin, you can try to use ruby filter to code it yourself. So, for the next time you can try it yourself. Enjoy it.

Hope this can help you.