Search code examples
elasticsearchfilebeat

Dynamic index in elasticsearch with filebeat


I have installed filebeat on a server and monitoring the below log files

C:\appl\dev\*\*\ms\*\*.log

I have configured filebeat.yml as below

filebeat.inputs:
    - paths:
        - 'C:\appl\dev\*\*\ms\*\*.log'
      type: log
processors:
    - dissect:
        field: log.file.path
        tokenizer: "C:\\appl\\dev\\%{field1}\\%{field2}\\ms\\"

I have two requirements,
one is to get the events with two additional fields filed1 and field2 (this is working as per the above configuration).
The other one is to have the index name in elastic appended with this additional field (field1 or field2).

Currently index is a variable (index: appname-%{[agent.version]}-%{+yyyy.MM.dd})
I need it to be appname-field1-%{[agent.version]}-%{+yyyy.MM.dd}

Please advice


Solution

  • For me adding output.elasticsearch wasn't enough, had to add setup.template as well

    output.elasticsearch:
      hosts: ["http://localhost:9200"]
      index: "appname-%{[field1]}-%{[agent.version]}-%{+yyyy.MM.dd}"
    
    setup.template:
      name: 'appname'
      pattern: 'appname-*'
      enabled: false