Filebeat send mulltiline postgres log as one log to filebeat set only to this specific source

Question

For example i have some sql log:

< 2019-03-13 09:50:50.431 CET >WYRAŻENIE:  SELECT
                              SELECT column1, column2, ...
FROM table_name
ORDER BY column1, column2, ... ASC|DESC;
< 2019-03-13 09:58:50.943 CET >UWAGA:  detail: RUCH_KRADZ, 0.05, sum: 0.25, date: 2019-03-03

In kibana each line is a seperate log. In filebeat i have:

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /opt/tomcat/logs/*.json
    - /var/lib/psql/logs/*

I want that only for the /var/lib/psql/logs/* the log should be as one beetween date. So in the example above we should have 2 logs in Kibana, not 5 - that is number of lines.

Answer 1

In filebeat configuration you can define multiple input sections each sections can have its own options

multiline.pattern Specifies the regular expression pattern to match, Where the first line of the message begins with a bracket (<)

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /opt/tomcat/logs/*.json

- type: log
  enabled: true
  paths:
    - /var/lib/psql/logs/*
  multiline.pattern: '^<'
  multiline.negate: true
  multiline.match: after

Check here for more details about Manage multiline messages