How to set kibana index pattern from filebeat?

Question

I am using elk stack with a node application. I am sending logs from host to logstash with filebeat, logsstash formats and send data to elastic and kibana reads from elastic. In kibana i see default index pattern like filebeat-2019.06.16.

I want to change this to application-name-filebeat-2019.06.16. But it's not working. I am looking for a way to do it in filebeat since there will be multiple applications/filebeats but one single logstash/elasticsearch/kibana.

I have tried this filebeat configs at filebeat.yml.

filebeat.inputs:
- type: log
  paths:
    - /var/log/*.log
  fields:
    - app_name: myapp

output.logstash:
  index: "%{fields.app_name}-filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  hosts: ["${ELK_ENDPOINT}"]
  ssl.enabled: true
  ssl:
    certificate_authorities:
      - /etc/pki/tls/certs/logstash-beats.crt

setup.template.name: "%{fields.app_name}-filebeat-%{[agent.version]}"

same kind of file will be with each of node appication host and filebeat.

also logstash is initialized with this configs

02-beats-input.conf

input {
  beats {
    port => 5044
    codec => "json"
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/logstash-beats.crt"
    ssl_key => "/etc/pki/tls/private/logstash-beats.key"
  }
}

30-output.conf

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => ["localhost"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

it is genarating index pattern like filebeat-2019.06.16. I want something like application-name-filebeat-2019.06.16.

Answer 1

You are sending your filebeat logs to logstash, you need to define the index name in the logstash pipeline, not in the filebeat config file.

Try the following output:

output {
  elasticsearch {
    hosts => ["localhost"]
    manage_template => false
    index => "%{[fields][app_name]}-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

To set the index name on filebeat you would need to send the logs directly to elasticsearch.

If you have other beats sending data to the same port and some of them do not have the field [fields][app_name] you could use a conditional on your output or create the field on your pipeline.