How to get specific namespace for log fluentd

Question

So I wanted to fetch logs from a specific namespace and send them to opensearch, so I didn't get a clue so I decided to try to make it like this:

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    k8s-app: fluentd-logging
  name: simple-fluentd-configmap
  namespace: fluent-log
data:
  fluent.conf: |
    <match fluent.**>  
      @type null   
    </match>     
    <source> 
      @type tail
      path /var/log/pods/containers*.log
      pos_file /var/log/containers.pos
      format none
    <filter **>
        @type record_modifier
        remove_keys "container_id,source"
        @type parser
        key_name log
        hash_value_field log
        <parse>
          @type json
        </parse>
    </filter>
    <match kubernetes.var.log.containers.**access**.log>
    @type opensearch
    port 443
    logstash_format false
    scheme https
    ssl_verify false
    </match>
    <match kubernetes.var.log.containers.**balance**.log>
    @type opensearch
    port 443
    logstash_format false
    scheme https
    ssl_verify false
    </match>

So the question is how to make it send specific logs based on namespace from kubernetes?

Answer 1

The second match clause in your configuration (the one starting with kubernetes.var.log...) should contain the namespace, and therefore you can filter based on specific namespaces and decide how to handle those specific logs.

If, for any reason, the log path in your cluster does not contain the namespace in its path, you can also use the kubernetes plugin.
It will enrich your logs with metadata relevant to the cluster, and allow you to extract the namespace logs originated from and deal with them accordingly.