Search code examples
dockerlogstashfilebeat

Filebeat multiline patter

I would like to configure a multiline pattern for each docker container that are deployed. I know that I can configure different filebeat inputs but the thing is that I don't know which container I am using because the path of the container log is like /var/lib/docker/containers/{id}/[{id}.log

Any ideas?

Solution

  • You can use glob-patterns in your filebeat configuration:

    a setting like this

    /var/lib/docker/containers/*/*.log

    Should match any file you'd be looking for?

    https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#input-paths

    Please make sure that a file is not being matched by multiple path-settings.

    Edit below as per added requirements.

    So for example you would have these 2 containers running:

    CONTAINER ID        IMAGE                                      COMMAND                  CREATED             STATUS              PORTS                    NAMES
77e87b8e772e        yadayada                                   "/hihihi"                 2 weeks ago        Up 19 seconds       0.0.0.0:9080->9080/tcp   container1
99e87b8e772e        blablabla                                  "/hahaha"                 2 weeks ago        Up 19 seconds       0.0.0.0:9080->9080/tcp   container2

    based on the provided information, the assumption is then that:

    container1 logs in /var/lib/docker/containers/77e87b8e772e/77e87b8e772e.log
    container2 logs in /var/lib/docker/containers/99e87b8e772e/99e87b8e772e.log

    This might be the config:

    filebeat.inputs:
- type: log
  paths: /var/lib/docker/containers/${CONTAINERID1}/${CONTAINERID1}.log
  multiline.pattern: '^=[A-Z]+|^$'
  multiline.negate: true
  multiline.match: after
- type: log
  paths: /var/lib/docker/containers/${CONTAINERID2}/${CONTAINERID2}.log
  multiline.pattern: '^=[1-9]+|^$'
  multiline.negate: true
  multiline.match: after

    So when starting filebeat, you do some additional things before actually running filebeat:

    export CONTAINERID1=$(docker ps|grep "container1$" | cut -d ' ' -f1)
export CONTAINERID2=$(docker ps|grep "container2$" | cut -d ' ' -f1)
./filebeat

    This way, as long and the container name remains the same, the ID can be different and will still work. Please note though that when you spin up a new (version of a) container, you will have to restart Filebeat to pick up the new path.

    Also please note that if you run Filebeat in a docker container itself, exporting the variable will most likely not be enough, you will have to edit the file using sed or something before you pass it into the filebeat-container