Some confusion here where I have to use filebeat multiline pattern to collec data. Question is how to use multiple pattern ? Here what i use now
multiline.pattern : '^Select'
So for above pattern we can see all word start from select will be match. So my question how about INSERT,UPDATE and DELETE word ?
Also one question can I use below pattern to indicate end of multiline match ?
multiline.flush_pattern: ';'
Any idea or help is highly appreciated
To your first question:
You can specify multiple words for the beginning of the message within a single regex. So if I understood you correctly, you want to include all log lines that start with Select, INSERT, UPDATE and DELETE. To achieve this you would define a group of valid values like so:
multiline.pattern : '^(Select|INSERT|UPDATE|DELETE)
The pipe-character ( | ) acts as an OR-Operator. Please note that by default regex is case sensitive. So e.g. messages that start with an uppercase SELECT would be ignored in the sample above.
To your second question:
Besides multiline.pattern you have to specify the settings multiline.match and multiline.negate:
multiline.match determines if the log lines before or after the pattern should be put into a single event.
multiline.negate determines if the following lines have to match the pattern.
So instead of specifying a particular end-character you tell Filebeat that every log line that matches the pattern AND is following that line should get aggregated UNTIL the following line matches again the pattern.
(See https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html for a full reference and description).
Example:
Assuming your log file is structured as following:
Select foo from bar\n where baz = 1\n and id =4711;\n\n
DELETE from bar\n where baz = null;\n\n
INSERT ...
the following config should do the job:
multiline.pattern : '^(Select|INSERT|UPDATE|DELETE)'
multiline.match: after
multiline.negate: true
I hope I could help you.