kubelet does not create symlinks to /var/log/containers

Question

I am trying to set up EFK stack on my k8s cluster using ansible repo.

When i tried to browse kibana dashboard it shows me next output:

After making some research, i found out that i don't have any log detected by Fluentd. I am running k8s 1.2.4 on minions and 1.2.0 on master. What i succeeded to understand, is that kubelet creates /var/log/containers directory, and make symlinks from all containers running in the cluster into it. After that Fluentd mounts share /var/log volume from the minion and have eventually access to all logs containers. So , it can send these logs to elastic search.

In my case i had /var/log/containers created, but it is empty, even /var/lib/docker/containers does not contain any log file. I used to use the following controllers and services for EFK stack setup:

es-controller.yaml

apiVersion: v1 kind: ReplicationController metadata: name: elasticsearch-logging-v1 namespace: kube-system labels: k8s-app: elasticsearch-logging version: v1 kubernetes.io/cluster-service: "true" spec: replicas: 2 selector: k8s-app: elasticsearch-logging version: v1 template: metadata: labels: k8s-app: elasticsearch-logging version: v1 kubernetes.io/cluster-service: "true" spec: containers: - image: gcr.io/google_containers/elasticsearch:v2.4.1 name: elasticsearch-logging resources: # need more cpu upon initialization, therefore burstable class limits: cpu: 1000m requests: cpu: 100m ports: - containerPort: 9200 name: db protocol: TCP - containerPort: 9300 name: transport protocol: TCP volumeMounts: - name: es-persistent-storage mountPath: /data env: - name: "NAMESPACE" valueFrom: fieldRef: fieldPath: metadata.namespace volumes: - name: es-persistent-storage emptyDir: {}

es-service.yaml

apiVersion: v1 kind: Service metadata: name: elasticsearch-logging namespace: kube-system labels: k8s-app: elasticsearch-logging kubernetes.io/cluster-service: "true" kubernetes.io/name: "Elasticsearch" spec: ports: - port: 9200 protocol: TCP targetPort: db selector: k8s-app: elasticsearch-logging

fluentd-es.yaml

apiVersion: v1 kind: Pod metadata: name: fluentd-es-v1.20 namespace: kube-system labels: k8s-app: fluentd-es version: v1.20 spec: containers: - name: fluentd-es image: gcr.io/google_containers/fluentd-elasticsearch:1.20 command: - '/bin/sh' - '-c' - '/usr/sbin/td-agent 2>&1 >> /var/log/fluentd.log' resources: limits: cpu: 100m volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers

kibana-controller.yaml

apiVersion: extensions/v1beta1 kind: Deployment metadata: name: kibana-logging namespace: kube-system labels: k8s-app: kibana-logging kubernetes.io/cluster-service: "true" spec: replicas: 1 selector: matchLabels: k8s-app: kibana-logging template: metadata: labels: k8s-app: kibana-logging spec: containers: - name: kibana-logging image: gcr.io/google_containers/kibana:v4.6.1 resources: # keep request = limit to keep this container in guaranteed class limits: cpu: 100m requests: cpu: 100m env: - name: "ELASTICSEARCH_URL" value: "http://elasticsearch-logging:9200" ports: - containerPort: 5601 name: ui protocol: TCP

kibana-service.yaml

apiVersion: v1 kind: Service metadata: name: kibana-logging namespace: kube-system labels: k8s-app: kibana-logging kubernetes.io/cluster-service: "true" kubernetes.io/name: "Kibana" spec: type: NodePort ports: - port: 5601 protocol: TCP targetPort: ui selector: k8s-app: kibana-logging

update:

I changed fluentd-es.yaml as following:

apiVersion: v1 kind: Pod metadata: name: fluentd-elasticsearch namespace: kube-system labels: k8s-app: fluentd-logging spec: containers: - name: fluentd-elasticsearch image: gcr.io/google_containers/fluentd-elasticsearch:1.15 resources: limits: memory: 200Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers

But when i run a pod "named gateway", i got in the fluentd log the next error: /var/log/containers/gateway-c3cuu_default_gateway-d5966a86e7cb1519329272a0b900182be81f55524227db2f524e6e23cd75ba04.log unreadable. It is excluded and would be examined next time.

Answer 1

Finally i found out what was causing the issue. when installing docker from CentOS 7 repo, there is an option (--log-driver=journald) which force docker to run log output to journald. The default behavior is to write these logs to json.log files.So, the only thing i had to do, delete the last mentioned option from /etc/sysconfig/docker.