I'm taking my first steps with elastic-stack with a practical approach, trying to make it work with an appliacation in my enviroment. I'm having difficulties understanding from scratch how to write grok filters. I would like to have one like this one working, so from that one, I can work the rest of them.
I've taken some udemy courses, I'm reading this "Elastic Stack 6.0", I'm reading the documentation, but I can't find a way to make this work as intended.
So far, the only grok filter I'm using that actually works, is as simple as (/etc/logstash/config.d/beats.conf)
input {
beats {
port => 5044
}
}
filter {
grok {
match => { 'message' => "%{DATE:date} %{TIME:time} %
{LOGLEVEL:loglevel}"
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
This is one of the log entries I'll need to work with, but there are many with different forms. I just need to have this one sorted out so I can adapt the filters to the rest.
2019-02-05 19:13:04,394 INFO [qtp1286783232-574:http://localhost:8080/service/soap/AuthRequest] [[email protected];oip=172.16.1.69;ua=zclient/8.8.9_GA_3019;soapId=3bde7ed0;] SoapEngine - handler exception: authentication failed for [admin], invalid password
I'd like to have this info, only when there is a "soapId" and when the field next to "INFO" starts with "qtq":
date: 2019-02-05
time: 19:13:04,394
loglevel: INFO
identifier: qtp1286783232-574
soap: http://localhost:8080/service/soap/AuthRequest
Which could also end in things like "GetInfoRequest" or "NoOpRequest"
account: [email protected]
oip: 172.16.1.69
client: zclient/8.8.9_GA_3019
soapid: 3bde7ed0
error: true (if either "invalid password" or "authentication failed" are found in the line)
If the conditions are not met, then I will apply other filters (which hopefully I will be able to write adapting this one as a base).
You can't have false in the output if you have invalid password in the input. You can only match what is there in the string.
I think you may use
%{DATE:date} %{TIME:time} %{LOGLEVEL:loglevel} *\[(?<identifier>qtp[^\]\[:]*):(?<soap>[^\]\[]*)]\s*\[name=(?<account>[^;]+);oip=(?<oip>[0-9.]+);ua=(?<client>[^;]+);soapId=(?<soapId>[^;]+);].*?(?:(?<error>authentication failed).*)?$
Here are the details of the added patterns:
* - 0+ spaces\[ - a [ char(?<identifier>qtp[^\]\[:]*) - Named group "identifier": qtp and then 0+ chars other than :, ] and [: - a colon(?<soap>[^\]\[]*) - Named group "soap": 0+ chars other than ] and []\s*\[name= - a ], then 0+ whitespaces and [name= substring(?<account>[^;]+) - Named group "account": 1+ chars other than ;;oip= - a literal substring(?<oip>[0-9.]+) - Named group "oip": 1+ digits and/or dots;ua= - a literal substring(?<client>[^;]+) - Named group "client": 1+ chars other than ;;soapId= - a literal substring(?<soapId>[^;]+) - Named group "soapId": 1+ chars other than ;;] - a literal substring.*? - any 0+ chars other than line break chars, as few as possible(?:(?<error>authentication failed).*)? - an optional group matching 1 or 0 occurrences of
authentication failed substring .* - all the rest of the line$ - end of input.