prometheus-operator includes DaemonSet to deploy node-exporter to every node in cluster. It works, but we lack some useful label information, for example:
# kubectl get nodes --all-namespaces --show-labels=true
NAME STATUS ROLES AGE VERSION LABELS
ip-1 Ready master 2d v1.10.12 ...,kubernetes.io/role=master,...
=
ip-2 Ready node 2d v1.10.12 ...,kubernetes.io/role=node,...
So we have useful info in labels - ip-1 is master, ip-2 is worker etc.
BUT this information is lost on node-exporter targets, because node labels are not propagated to daemonset node-exporter pods.
So in prometheus, I can't group nodes by their type, for example.
Maybe there's a way how to achieve this? Thanks!
It seems that you need to use relabel_config
This is an example: https://www.robustperception.io/automatically-monitoring-ec2-instances
P.S. As for roles specific, you can find usefull this post as well: How to have labels for machine roles
Update: To get other node details, not available from metadata, sidecar or init container can be used, for example: init container for node properties example.
Also, it is open issue to make node labels available for pod: 40610