I'm trying to configure Fluentbit in Kubernetes to get Logs from application PODs/Docker Containers and send this log messages to Graylog using GELF format, but this is not working.
See my stack below:
The problem is the fluentbit not read the log from systemd I'm not get any log in both outputs(Systemd,Stdout), the STDOUT is just to help in troubleshooting.
I don't know why I'm not able to read from systemd. I followed the documentation exactly https://docs.fluentbit.io/manual/input/systemd
My K8S configurations:
fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: log
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level debug
Daemon off
@INCLUDE input-systemd.conf
@INCLUDE output-stdout.conf
input-systemd.conf: |
[INPUT]
Name systemd
Tag host.*
Parser json
Systemd_Filter _SYSTEMD_UNIT=docker.service
output-graylog.conf: |
[OUTPUT]
Name gelf
Match *
Host 10.142.15.214
Port 12201
Mode tcp
Gelf_Short_Message_Key log
output-stdout.conf: |
[OUTPUT]
Name stdout
Match *
fluent-bit-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: log
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.3.5
imagePullPolicy: Always
ports:
- containerPort: 2020
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
fluent-bit-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: log
fluent-bit-role.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
verbs: ["get", "list", "watch"]
fluent-bit-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: log
My Fluentbit OUTPUT(STDOUT) just for debug:
$ kubectl logs -f fluent-bit-2bzxb -n log
[2020/02/20 18:54:23] [Warning] [config] I cannot open /fluent-bit/etc/..2020_02_20_18_54_22.252769193/parsers_custom.conf file
[2020/02/20 18:54:23] [ info] [storage] initializing...
[2020/02/20 18:54:23] [ info] [storage] in-memory
[2020/02/20 18:54:23] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/02/20 18:54:23] [ info] [engine] started (pid=1)
[2020/02/20 18:54:23] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/02/20 18:54:23] [ info] [filter_kube] local POD info OK
[2020/02/20 18:54:23] [ info] [filter_kube] testing connectivity with API server...
[2020/02/20 18:54:23] [ info] [filter_kube] API server connectivity OK
[2020/02/20 18:54:23] [ info] [sp] stream processor started
The problem is I'm not getting any log from systemd with this configuration
Thank you @edsiper I fix my Daemonset adding "path: /run/log"
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: log
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.3.5
imagePullPolicy: Always
ports:
- containerPort: 2020
env:
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: systemdlog
mountPath: /run/log
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: systemdlog
hostPath:
path: /run/log
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"