This is my config:
input{
beats {
port => 55556
}
}
filter {
if "iis" in [tags] {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:S-SiteName} %{WORD:S-ComputerName} %{IPORHOST:S-IP} %{WORD:CS-Method} %{URIPATH:CS-URI-Stem} %{NOTSPACE:CS-URI-QUERY} %{NUMBER:S-Port}$ %{NUMBER:SC-Win32-Status} %{NUMBER:SC-Bytes} %{NUMBER:CS-Bytes} %{NUMBER:Time-Taken}"}
}
}
}
filter {
if "nx" in [tags] {
grok {
match => { "message" => "\[%{TIMESTAMP_ISO8601:log_timestamp}\] (?<LogLevel>\[\w+\s*\]) (?<thread>\[\s*\d*\]) (?<snName>\[\w*\]) (?<snId>\[\d*\s*\d*\]) %{GREEDYDATA:message}"}
}
}
}
output{
if "nx" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "nx-%{+YYYY.ww}"
user => "user"
password => "pass"
}
}
if "iis" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "iis-%{+YYYY.ww}"
user => "user"
password => "pass"
}
}
}
Here's a log sample:
[2018-02-18 15:19:04.668] [INFO ] [ 155] [AliveReportCommand] [875076019 53033] - ProcessRequest Ended: elapsed time=00:00:00.0967851, _parser.Device.IsSuccess=False
[2018-02-18 15:25:32.716] [DEBUG] [ 181] [] [] - Web Facade called: streamIDParam=-1
This log corresponds to the "nx" logs. For some reason it just does not get ingested. I ran grok simulation using the grok above and that code and it worked out great. However, it's like the "nx" filter is completely ignored. Even if I eliminate the "IIS" filter, still no info would show up on Kibana. Also no errors whatsoever in Logstash. I have the same setup for both using Filebeat:
- type: log
enabled: true
paths: c:\logs\*\*.log
exclude_files: ['mybeat.*']
tags: ["nx"]
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
- type: log
enabled: true
paths: C:\inetpub\logs\LogFiles\*\*.log
tags: ["iis"]
I feel so embarrassed...
The problem wasnt with the filter to begin with. It was with the timestamp set on Kibana. After selecting the right field it showed up immediately.