I have several applications running on a single server. I'd like to use filebeat to ship the logs of each of them to logstash. However, for the sake of configuration management, I'd like to be able to add configuration to filebeat for each app separately.
Logstash reads its config from a conf.d directory. It's my understanding that one can add files there and they get combined when logstash loads them. Is there any similar feature for filebeat? Or am I stuck with maintaining a single filebeat.yml file per server?
I'm running both filebeat and logstash as services on CentOS 7, using the yum/rpm packages from elastic's repositories. Filebeat is version 1.3.1 and logstash is version 2.4.0.
Yes, Filebeat has a conf.d like feature, but it is not enabled by default. Filebeat will look inside of the declared directory for additional *.yml files that contain prospector configurations. The configuration varies by Filebeat major version.
The behavior is the same as 6.x, but the config option is filebeat.config.inputs instead of filebeat.config.prospectors.
# /etc/filebeat/filebeat.yml
filebeat.config.inputs:
enabled: true
path: inputs.d/*.yml
Then create individual config files for each app that's generating logs.
# /etc/filebeat/inputs.d/someapp.yml
- paths:
- /var/log/someapp/stdout.log
fields:
app: someapp
You specify a path option in the filebeat.config.prospectors section of the filebeat.yml file.
filebeat.config.prospectors:
enabled: true
path: /etc/filebeat/conf.d/*.yml
/etc/filebeat/conf.d/someapp.yml
Note that this file does not contain filebeat.prospectors like it did in earlier versions.
- paths:
- /var/log/someapp/stdout.log
fields:
app: someapp
You declare the directory inside of the main filebeat.yml using the config_dir option.
filebeat:
config_dir: /etc/filebeat/conf.d
/etc/filebeat/conf.d/someapp.yml
filebeat:
prospectors:
- paths:
- /var/log/someapp/stdout.log
fields:
app: someapp