Why bitnami Fluentd daemonsets dont generate logs as standard output?

Question

I have deployed the Bitnami EFK helm chart on the K8s cluster. https://github.com/bitnami/charts/tree/master/bitnami/fluentd

All pod runs fine but Fluentd not showing any logs. I don't know if I have something missing in the config. However, the cluster is restricted, and don't know if that makes any difference. I deployed the same EFK on the unrestricted cluster with the same configuration and works totally fine.

kkot@ltp-str-00-0085:~/logging-int$ kk get pod
NAME                                                             READY   STATUS    RESTARTS   AGE
elasticsearch-elasticsearch-coordinating-only-5f5656cdd5-9d4lj   1/1     Running   0          6h34m
elasticsearch-elasticsearch-coordinating-only-5f5656cdd5-h6lbd   1/1     Running   0          6h34m
elasticsearch-elasticsearch-data-0                               1/1     Running   0          6h34m
elasticsearch-elasticsearch-data-1                               1/1     Running   0          6h34m
elasticsearch-elasticsearch-master-0                             1/1     Running   0          6h34m
elasticsearch-elasticsearch-master-1                             1/1     Running   0          6h34m
fluentd-0                                                        1/1     Running   0          6h10m
fluentd-4glgs                                                    1/1     Running   2          6h10m
fluentd-59tzz                                                    1/1     Running   0          5h43m
fluentd-b8bc8                                                    1/1     Running   2          6h10m
fluentd-qfdcs                                                    1/1     Running   2          6h10m
fluentd-sf2hk                                                    1/1     Running   2          6h10m
fluentd-trvwx                                                    1/1     Running   0          95s
fluentd-tzqw8                                                    1/1     Running   2          6h10m
kibana-656d55f94d-8qf8f                                          1/1     Running   0          6h28m



kkot@ltp-str-00-0085:~/logging-int$ kk logs fluentd-qfdcs

Error Log:

2021-02-24 10:52:15 +0000 [warn]: #0 pattern not matched: "{\"log\":\"2021-02-24 10:52:13 +0000 [warn]: #0 pattern not matched: \\"{\\\\"log\\\\":\\\\"

Has anyone faced the same issue? Thanks

Answer 1

Could you please share what configuration is your forwarder using?

In the latest version of the chart (3.6.2) it will use the following by default:

  configMapFiles:
    fluentd.conf: |
      # Ignore fluentd own events
      <match fluent.**>
        @type null
      </match>
      @include fluentd-inputs.conf
      @include fluentd-output.conf
      {{- if .Values.metrics.enabled }}
      @include metrics.conf
      {{- end }}
    fluentd-inputs.conf: |
      # HTTP input for the liveness and readiness probes
      <source>
        @type http
        port 9880
      </source>
      # Get the logs from the containers running in the node
      <source>
        @type tail
        path /var/log/containers/*.log
        # exclude Fluentd logs
        exclude_path /var/log/containers/*fluentd*.log
        pos_file /opt/bitnami/fluentd/logs/buffers/fluentd-docker.pos
        tag kubernetes.*
        read_from_head true
        <parse>
          @type json
        </parse>
      </source>
      # enrich with kubernetes metadata
      <filter kubernetes.**>
        @type kubernetes_metadata
      </filter>

By the error log you shared:

2021-02-24 10:52:15 +0000 [warn]: #0 pattern not matched: "{\"log\":\"2021-02-24 10:52:13 +0000 [warn]: #0 pattern not matched: \\"{\\\\"log\\\\":\\\\"

I notice two things:

• The fluentd pods seem to be collecting their own logs, which shouldn't be happening because of:

     # exclude Fluentd logs
     exclude_path /var/log/containers/*fluentd*.log
  

• The JSON logs are not being parsed although it is configured:

      <parse>
         @type json
      </parse>
  

Maybe you have omitted the configMapFiles in your values.yaml?