Why mounted hostPath doesn't work on kubernetes of GKE

Question

I deployed these 2 kinds of services on GKE. Just want to confirm if the nginx data been mounted to the host.

Yaml

Nginx deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: beats
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80
        volumeMounts:
        - name: nginx-data
          mountPath: /var/log/nginx
      volumes:
      - name: nginx-data
        hostPath:
          path: /var/lib/nginx-data
          type: DirectoryOrCreate

Filebeat

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: beats
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.modules:
      - module: nginx

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          hints.enabled: false
          templates:
            - condition.contains:
                kubernetes.namespace: beats
              config:
                - module: nginx
                  access:
                    enabled: true
                    var.paths: ["/var/lib/nginx-data/access.log*"]
                    subPath: access.log
                    tags: ["access"]
                  error:
                    enabled: true
                    var.paths: ["/var/lib/nginx-data/error.log*"]
                    subPath: error.log
                    tags: ["error"]

    output.logstash:
      hosts: ["logstash.beats.svc.cluster.local:5044"]

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: beats
  labels:
    k8s-app: filebeat
spec:
  selector:
    matchLabels:
      k8s-app: filebeat
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
        - name: filebeat
          image: docker.elastic.co/beats/filebeat:7.10.0
          args: [
            "-c", "/etc/filebeat.yml",
            "-e",
          ]
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
            - name: config
              mountPath: /etc/filebeat.yml
              subPath: filebeat.yml
              readOnly: true
            - name: data
              mountPath: /usr/share/filebeat/data
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
            - name: nginx-data
              mountPath: /var/lib/nginx-data
      volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: filebeat-config
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
        - name: data
          hostPath:
            path: /var/lib/filebeat-data
            type: DirectoryOrCreate
        - name: nginx-data
          hostPath:
            path: /var/lib/nginx-data
            type: DirectoryOrCreate

Check deploy

Nginx

kubectl describe po nginx-658f45f77-dpflp -n beats

...
Volumes:
  nginx-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/nginx-data
    HostPathType:  DirectoryOrCreate

Filebeat pod

kubectl describe po filebeat-42wh7 -n beats

....
Volumes:
  ....
  nginx-data:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/nginx-data
    HostPathType:  DirectoryOrCreate

Check on nginx pod

# mount | grep nginx
/dev/sda1 on /var/log/nginx type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30)
/dev/sda1 on /var/cache/nginx type ext4 (rw,nosuid,nodev,relatime,commit=30)

root@nginx-658f45f77-dpflp:/# ls /var/log/nginx/
access.log  error.log

Check on filebeat pod

# mount | grep nginx
/dev/sda1 on /var/lib/nginx-data type ext4 (rw,nosuid,nodev,noexec,relatime,commit=30)

# ls /var/lib/nginx-data
(NULL)

---

The hostPath - /var/lib/nginx-data doesn't work. If use minikube, it can work. I can use minikube ssh to check the path on the host. But on GKE, how can I check the hostPath on remote machine?

Answer 1

On GKE (and other hosted Kubernetes offerings from public-cloud providers) you can't directly connect to the nodes. You'll have to confirm using debugging tools like kubectl exec that content is getting from one pod to the other; since you're running filebeat as a DaemonSet, you'll need to check the specific pod that's running on the same node as the nginx pod.

The standard Docker Hub nginx image is configured to send its logs to the container stdout/stderr (more specifically, absent a volume mount, /var/log/nginx/access.log is a symlink to /proc/self/stdout). In a Kubernetes environment, the base log collector setup you show will be able to collect its logs. I'd just delete the customizations you're asking about in this question – don't create a hostPath directory, don't mount anything over the container's /var/log/nginx, and don't have special-case log collection for this one pod.