Search code examples
elasticsearchkubernetesfluentd

separate index for different namespace fluentD


    <match pod.logs.kubernetes.**>
      @type elasticsearch
      index_name {{.Values.fluentdelasticsearch.indexName}}
      host {{.Values.fluentdelasticsearch.host}}
      port 443
      scheme https
      type_name doc
      time_key_format "%Y-%m-%dT%H:%M:%S.%N%z"
      time_key time
      include_timestamp true
      reconnect_on_error true
      reload_connections true
      reload_on_failure true
      resurrect_after 5
      <buffer>
        flush_interval 5
        chunk_limit_size 2m
        queued_chunks_limit_size 32
        retry_forever true
      </buffer>
      @log_level debug
    </match>

I have fluentD deployed in Cluster level, there are two namespaces dev & test in the cluster. Current logs for both pods are generated in same index, I need to have separate indices for namespaces.


Solution

  •     <filter pod.logs.kubernetes.**>
          @type kubernetes_metadata
        </filter>
    
    
       <match pod.logs.kubernetes.**>      
          @type elasticsearch_dynamic
          index_name ${record['kubernetes']['namespace_name']}-index
         ..... 
         ...... 
       </match>