Search code examples
logstashlogstash-grokelk

Grok Pattern to Extract Syslog in Linux(Ubuntu)

If you have any grok pattern to extract syslog in ubuntu please provide it. Thank You!

Edited --->>

My syslog example ->

"Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:23:02Z\",\"tags\":[],\"pid\":1877,\"method\":\"post\",\"statusCode\":200,\"req\":{\"url\":\"/internal/search/es\",\"method\":\"post\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"content-length\":\"861\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"origin\":\"http://example.com\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":65,\"contentLength\":9},\"message\":\"POST /internal/search/es 200 65ms - 9.0B\"}"

I tried following filter ->

filter {
      grok {
            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"}
            #match => {"syslog_message" => "%{WORD:FILTERED}" }
            #add_field => [ "received_at", "%{@timestamp}" ]
            #add_field => [ "received_from", "%{host}" ]
            remove_field => ["host","message"]
            
      }

      mutate{

            rename => ["@timestamp","time"]

      }
}

Then I got the below output. And I want to extract the syslog message part.

 "time" => 2020-08-20T11:17:57.995Z,
      "syslog_message" => "message repeated 9 times: [ {\"type\":\"response\",\"@timestamp\":\"2020-08-20T10:27:22Z\",\"tags\":[],\"pid\":1877,\"method\":\"get\",\"statusCode\":200,\"req\":{\"url\":\"/api/rollup/indices\",\"method\":\"get\",\"headers\":{\"connection\":\"upgrade\",\"host\":\"example.com\",\"kbn-version\":\"7.8.1\",\"user-agent\":\"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36\",\"content-type\":\"application/json\",\"accept\":\"*/*\",\"referer\":\"http://example.com/app/kibana\",\"accept-encoding\":\"gzip, deflate\",\"accept-language\":\"en-US,en;q=0.9,si;q=0.8\"},\"remoteAddress\":\"127.0.0.1\",\"userAgent\":\"127.0.0.1\",\"referer\":\"http://example.com/app/kibana\"},\"res\":{\"statusCode\":200,\"responseTime\":31,\"contentLength\":9},\"message\":\"GET /api/rollup/indices 200 31ms - 9.0B\"},]",
    "syslog_timestamp" => "Aug 20 15:58:52",
                "path" => "/var/log/syslog",
            "@version" => "1",
     "syslog_hostname" => "amantha-ubuntu-server",
      "syslog_program" => "amantha"
Solution

  • The first thing I'd do is use gsub on that line to remove the " and \.

    You might use:

    mutate {
 gsub => [ "message", "[\\\"]", "" ]
}

    This'll leave you with:

    Aug 20 15:53:02 amantha-ubuntu-server kibana[1877]: {type:response,@timestamp:2020-08-20T10:23:02Z,tags:[],pid:1877,method:post,statusCode:200,req:{url:/internal/search/es,method:post,headers:{connection:upgrade,host:example.com,content-length:861,kbn-version:7.8.1,user-agent:Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36,content-type:application/json,accept:*/*,origin:http://example.com,referer:http://example.com/app/kibana,accept-encoding:gzip, deflate,accept-language:en-US,en;q=0.9,si;q=0.8},remoteAddress:127.0.0.1,userAgent:127.0.0.1,referer:http://example.com/app/kibana},res:{statusCode:200,responseTime:65,contentLength:9},message:POST /internal/search/es 200 65ms - 9.0B}

    And you can use the following line to grab the "syslog message" after gsub. I don't really know if you wanted it broken down any further, but more than happy to help if that's desired.

    (?<syslog_timestamp>%{SYSLOGTIMESTAMP}) (?<syslog_hostname>%{SYSLOGHOST}) (?<syslog_program>%{SYSLOGPROG}): {(?<syslog_message>(?<={).*(?=}))}