I have installed kube-prometheus-stack as a dependency in my helm chart on a local Docker for Mac Kubernetes cluster v1.19.7.
The myrelease-name-prometheus-node-exporter service is failing with errors received from the node-exporter daemonset after installation of the helm chart for kube-prometheus-stack is installed. This is installed in a Docker Desktop for Mac Kubernetes Cluster environment.
release-name-prometheus-node-exporter daemonset error log
MountVolume.SetUp failed for volume "flaskapi-prometheus-node-exporter-token-zft28" : failed to sync secret cache: timed out waiting for the condition
Error: failed to start container "node-exporter": Error response from daemon: path / is mounted on / but it is not a shared or slave mount
Back-off restarting failed container
The scrape targets for kube-scheduler:http://192.168.65.4:10251/metrics, kube-proxy:http://192.168.65.4:10249/metrics, kube-etcd:http://192.168.65.4:2379/metrics, kube-controller-manager:http://192.168.65.4:10252/metrics and node-exporter:http://192.168.65.4:9100/metrics are marked as unhealthy. All show as connection refused, except for kube-etcd which displays connection reset by peer.
Chart.yaml
apiVersion: v2
appVersion: "0.0.1"
description: A Helm chart for flaskapi deployment
name: flaskapi
version: 0.0.1
dependencies:
- name: kube-prometheus-stack
version: "14.4.0"
repository: "https://prometheus-community.github.io/helm-charts"
- name: ingress-nginx
version: "3.25.0"
repository: "https://kubernetes.github.io/ingress-nginx"
- name: redis
version: "12.9.0"
repository: "https://charts.bitnami.com/bitnami"
Values.yaml
hostname: flaskapi-service
redis_host: flaskapi-redis-master.default.svc.cluster.local
redis_port: "6379"
Environment Mac OS Catalina 10.15.7 Docker Desktop For Mac 3.2.2(61853) with docker engine v20.10.5 Local Kubernetes 1.19.7 Cluster provided by Docker Desktop For Mac
Prometheus Operator version:
kube-prometheus-stack 14.4.0
Kubernetes version information:
kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:15:20Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/alertmanager-flaskapi-kube-prometheus-s-alertmanager-0 2/2 Running 0 16m
pod/flask-deployment-775fcf8ff-2hp9s 1/1 Running 0 16m
pod/flask-deployment-775fcf8ff-4qdjn 1/1 Running 0 16m
pod/flask-deployment-775fcf8ff-6bvmv 1/1 Running 0 16m
pod/flaskapi-grafana-6cb58f6656-77rqk 2/2 Running 0 16m
pod/flaskapi-ingress-nginx-controller-ccfc7b6df-qvl7d 1/1 Running 0 16m
pod/flaskapi-kube-prometheus-s-operator-69f4bcf865-tq4q2 1/1 Running 0 16m
pod/flaskapi-kube-state-metrics-67c7f5f854-hbr27 1/1 Running 0 16m
pod/flaskapi-prometheus-node-exporter-7hgnm 0/1 CrashLoopBackOff 8 16m
pod/flaskapi-redis-master-0 1/1 Running 0 16m
pod/flaskapi-redis-slave-0 1/1 Running 0 16m
pod/flaskapi-redis-slave-1 1/1 Running 0 15m
pod/prometheus-flaskapi-kube-prometheus-s-prometheus-0 2/2 Running 0 16m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 16m
service/flask-api-service ClusterIP 10.108.242.86 <none> 4444/TCP 16m
service/flaskapi-grafana ClusterIP 10.98.186.112 <none> 80/TCP 16m
service/flaskapi-ingress-nginx-controller LoadBalancer 10.102.217.51 localhost 80:30347/TCP,443:31422/TCP 16m
service/flaskapi-ingress-nginx-controller-admission ClusterIP 10.99.21.136 <none> 443/TCP 16m
service/flaskapi-kube-prometheus-s-alertmanager ClusterIP 10.107.215.73 <none> 9093/TCP 16m
service/flaskapi-kube-prometheus-s-operator ClusterIP 10.107.162.227 <none> 443/TCP 16m
service/flaskapi-kube-prometheus-s-prometheus ClusterIP 10.96.168.75 <none> 9090/TCP 16m
service/flaskapi-kube-state-metrics ClusterIP 10.100.118.21 <none> 8080/TCP 16m
service/flaskapi-prometheus-node-exporter ClusterIP 10.97.61.162 <none> 9100/TCP 16m
service/flaskapi-redis-headless ClusterIP None <none> 6379/TCP 16m
service/flaskapi-redis-master ClusterIP 10.96.192.160 <none> 6379/TCP 16m
service/flaskapi-redis-slave ClusterIP 10.107.119.108 <none> 6379/TCP 16m
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d1h
service/prometheus-operated ClusterIP None <none> 9090/TCP 16m
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/flaskapi-prometheus-node-exporter 1 1 0 1 0 <none> 16m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/flask-deployment 3/3 3 3 16m
deployment.apps/flaskapi-grafana 1/1 1 1 16m
deployment.apps/flaskapi-ingress-nginx-controller 1/1 1 1 16m
deployment.apps/flaskapi-kube-prometheus-s-operator 1/1 1 1 16m
deployment.apps/flaskapi-kube-state-metrics 1/1 1 1 16m
NAME DESIRED CURRENT READY AGE
replicaset.apps/flask-deployment-775fcf8ff 3 3 3 16m
replicaset.apps/flaskapi-grafana-6cb58f6656 1 1 1 16m
replicaset.apps/flaskapi-ingress-nginx-controller-ccfc7b6df 1 1 1 16m
replicaset.apps/flaskapi-kube-prometheus-s-operator-69f4bcf865 1 1 1 16m
replicaset.apps/flaskapi-kube-state-metrics-67c7f5f854 1 1 1 16m
NAME READY AGE
statefulset.apps/alertmanager-flaskapi-kube-prometheus-s-alertmanager 1/1 16m
statefulset.apps/flaskapi-redis-master 1/1 16m
statefulset.apps/flaskapi-redis-slave 2/2 16m
statefulset.apps/prometheus-flaskapi-kube-prometheus-s-prometheus 1/1 16m
kubectl get svc -n kube-system
flaskapi-kube-prometheus-s-coredns ClusterIP None <none> 9153/TCP 29s
flaskapi-kube-prometheus-s-kube-controller-manager ClusterIP None <none> 10252/TCP 29s
flaskapi-kube-prometheus-s-kube-etcd ClusterIP None <none> 2379/TCP 29s
flaskapi-kube-prometheus-s-kube-proxy ClusterIP None <none> 10249/TCP 29s
flaskapi-kube-prometheus-s-kube-scheduler ClusterIP None <none> 10251/TCP 29s
flaskapi-kube-prometheus-s-kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 2d18h
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 5d18h
Tried updating values.yaml to include this:
Updated values.yaml
prometheus-node-exporter:
hostRootFsMount: false
and this:
prometheus:
prometheus-node-exporter:
hostRootFsMount: false
However, the issue described, remains, except the log for node-exporter daemonset now gives:
failed to try resolving symlinks in path "/var/log/pods/default_flaskapi-prometheus-node-exporter-p5cc8_54c20fc6-c914-4cc6-b441-07b68cda140e/node-exporter/3.log": lstat /var/log/pods/default_flaskapi-prometheus-node-exporter-p5cc8_54c20fc6-c914-4cc6-b441-07b68cda140e/node-exporter/3.log: no such file or directory
Updated Information From Comment Suggestions
kubectl get pod flaskapi-prometheus-node-exporter-p5cc8
No args available since node-exporter crashing...
NAME READY STATUS RESTARTS AGE
flaskapi-prometheus-node-exporter-p5cc8 0/1 CrashLoopBackOff 7 14m
The Args from the yaml output of kubectl describe pod flaskapi-prometheus-node-exporter-p5cc8 gives:
Args:
--path.procfs=/host/proc
--path.sysfs=/host/sys
--path.rootfs=/host/root
--web.listen-address=$(HOST_IP):9100
--collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)
--collector.filesystem.ignored-fs-types=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
After updating the values.yaml to include root kube-prometheus-stack as suggested in comments of answer allows the prometheus-node-exporter daemonset to start successfully. However, the scrape targets mentioned above are still unavailable....
kube-prometheus-stack:
prometheus-node-exporter:
hostRootFsMount: false
How do I get the node-exporter working and make the associated scrape targets healthy?
Is the node-exporter of the kube-prometheus-stack helm chart incompatible with Docker Desktop for Mac Kubernetes clusters?
I have raised this as an issue at kube-prometheus-stack with log output included for scrape targets for docker-desktop and minikube clusters.
Conclusion It looks as though the unavailable scrape targets is a problem/bug with kube-prometheus-stack. I searched and found similar issues on their GitHub page: 713 and 718. Tried on a minikube cluster with hyperkit vm-driver. On minikube the node-exporter functions out of the box, but the scrape targets issue still occurs. Not sure what a safe solution is?
I may investigate an alternative helm chart dependency for prometheus and grafana...
This issue was solved recently. Here is more information: https://github.com/prometheus-community/helm-charts/issues/467 and here: https://github.com/prometheus-community/helm-charts/pull/757
Here is the solution (https://github.com/prometheus-community/helm-charts/issues/467#issuecomment-802642666):
[you need to] opt-out the rootfs host mount (preventing the crash). In order to do that you need to specify the following value in values.yaml file:
prometheus-node-exporter: hostRootFsMount: false