Search code examples
elasticsearchfilebeatilm

Is it possible to define two ILM policies in single filebeat.yml

I'd like to know if it is feasible to configure FileBeat in following way:

I have single rabbitmq with several MQTT topics. I'd like to save messages from topics foo/# in index index-foo and messages from topics bar/# in index index-bar in order to set up separate rolling policies.

I managed to split MQTT messages into separate indexes by setting index property on filebeat.inputs but I can't apply rolling policies then.

Moreover ILM syntax

setup.ilm:
  enabled: true
  rollover_alias: "jkk-logs-filebeat-yml-rollover_alias"
  pattern: "{now/d}-000001"
  overwrite: true
  policy_name: "jkk-policy"
  policy_file: /usr/share/filebeat/ilm.json

seams to allow to set up only one policy.

I'd like to avoid running two separate FileBeat instances.

Is it possible? Can anyone point me to a tutorial or GitHub repository which shows how to do that?

I'm using ES and FB version 7.17.0 (if it does matter).

Solution

  • there's no way to handle multiples like that and you will need to setup the ilm policies directly in Elasticsearch, and just tell Filebeat to send to the alias