Search code examples
logstashlogstash-grokfilebeat

Logstash ignores(?) one grok filter


This is my config:

input{

        beats {
                port => 55556

        }


}

filter {
  if "iis" in [tags] {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{WORD:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} %{NOTSPACE:CS-URI-QUERY} %{NUMBER:S-Port}$ %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
    }
  }
}

filter {
  if "nx" in [tags] {
    grok {
      match => { "message" => "\[%{TIMESTAMP_ISO8601:log_timestamp}\] (?<LogLevel>\[\w+\s*\]) (?<thread>\[\s*\d*\]) (?<snName>\[\w*\]) (?<snId>\[\d*\s*\d*\]) %{GREEDYDATA:message}"}
    }
  }
}



output{
        if "nx" in [tags] {

                elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "nx-%{+YYYY.ww}"
                        user => "user"
                        password => "pass"
                }
        }

        if "iis" in [tags] {

                elasticsearch {
                        hosts => ["localhost:9200"]
                        index => "iis-%{+YYYY.ww}"
                        user => "user"
                        password => "pass"
                }
        }

}

Here's a log sample:

[2018-02-18 15:19:04.668] [INFO ] [   155] [AliveReportCommand] [875076019 53033] - ProcessRequest Ended: elapsed time=00:00:00.0967851, _parser.Device.IsSuccess=False
[2018-02-18 15:25:32.716] [DEBUG] [   181] [] [] - Web Facade called: streamIDParam=-1

This log corresponds to the "nx" logs. For some reason it just does not get ingested. I ran grok simulation using the grok above and that code and it worked out great. However, it's like the "nx" filter is completely ignored. Even if I eliminate the "IIS" filter, still no info would show up on Kibana. Also no errors whatsoever in Logstash. I have the same setup for both using Filebeat:

- type: log
  enabled: true
  paths: c:\logs\*\*.log
  exclude_files: ['mybeat.*']
  tags: ["nx"]
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
- type: log
  enabled: true
  paths: C:\inetpub\logs\LogFiles\*\*.log
  tags: ["iis"]

Solution

  • I feel so embarrassed...

    The problem wasnt with the filter to begin with. It was with the timestamp set on Kibana. After selecting the right field it showed up immediately.