Search code examples
elasticsearchloggingkubernetesfilebeatamazon-eks

How to create indices based on kubernetes metadata

I am using filebeat in Kubernetes to ship logs to elastic search. I want to create indexes based on namespaces. I'm trying to create a custom index for my different apps in a Kubernetes cluster, but this not working. I used below conf:-

output.elasticsearch:

  index: "%{[kubernetes.labels.app]:filebeat}-%{[beat.version]}-%{+yyyy.MM.dd}"

Filebeat Kube-manifest link- https://github.com/anup1384/k8s-filebeat

Solution

  • Create a custom index using Kubernetes metadata. So here I'm creating an index based on pod name metadata.

    logstash_prefix ${record['kubernetes']['pod_name']}

    For more details:

    https://medium.com/faun/how-to-create-custom-indices-based-on-kubernetes-metadata-using-fluentd-beed062faa5d