Search code examples
kuberneteselastic-stackfilebeatelastic-cloudelastic-beats

Filebeat is not forwarding nginx ingress controller logs


Filebeat 7.12.1 ECK operator 2.2

I'm trying to setup the filbeat for the Nginx-ingress access logs in my ECK stack (installed in GKE). I can access the logs directly in the pod but nothing is coming to my Kibana dashboard. I have set up two filebeat.autodiscover.providers

  1. hints.enabled: true, which looks for all the containers with co.elastic.logs/enabled: "true"
  2. Checks the container containing name ingress. I can confirm that the name of the pod is nginx-ingress-ingress-nginx-controller-xxxx-xxxxx

Below is my Filebeat auto discover content:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
  namespace: search
spec:
  type: filebeat
  version: 7.12.1
  elasticsearchRef:
    name: elastic-search
  kibanaRef:
    name: kibana-web
  config:
    filebeat.autodiscover.providers:
    - node: ${NODE_NAME}
      type: kubernetes
      hints.enabled: true
      #add_resource_metadata.namespace.enabled: true
      hints.default_config.enabled: "false"
    - node: ${NODE_NAME}
      type: kubernetes
      #add_resource_metadata.namespace.enabled: true
      hints.default_config.enabled: "false"
      templates:
      - condition:
          contains: 
            kubernetes.container.name: ingress
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
          exclude_lines: ["^\\s+[\\-`('.|_]"]
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: filebeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        dnsPolicy: ClusterFirstWithHostNet
        #hostNetwork: true # Allows to provide richer host metadata
        containers:
        - name: filebeat
          securityContext:
            runAsUser: 0
            # If using Red Hat OpenShift uncomment this:
            #privileged: true
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          resources:
            requests:
              memory: 200Mi
              cpu: 0.2
            limits:
              memory: 300Mi
              cpu: 0.4
              
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers

Solution

  • Adding the answer here in case someone else run into this issue.

    The issue is how I'm checking the contains condition. It should've been kubernetes.pod.name instead of kubernetes.container.name. So I replaced

    - condition:
              contains: 
                kubernetes.container.name: ingress
    

    to

    - condition:
              contains: 
                kubernetes.pod.name: ingress
    

    in the above file and things started to work!