Kube-Prometheus-Stack and securing external node-exporter

Question

i would like to secure my external node-exporter with tls and authentication, so that in my network, not everyone is able to access the metrics exposed by the node-exporter.

on prometheus side i have service, Servicemonitor and endpoint:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
    prometheus.io/scrape: 'true'
  labels:
    app: node-exporter-vm-agent
    jobLabel: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  externalName: 192.168.1.72
  ports:
  - name: metrics
    port: 9100
    protocol: TCP
    targetPort: 9100
  selector:
    app: vm-agent
    release: prom00
  type: ExternalName

---

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
  labels:
    app: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  endpoints:
    - port: metrics
      scheme: https
      tlsConfig:
        insecureSkipVerify: true
  jobLabel: jobLabel
  selector:
    matchLabels:
      app: node-exporter-vm-agent
      release: prom00

---

apiVersion: v1
kind: Endpoints
metadata:
  labels:
    app: node-exporter-vm-agent
    jobLabel: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
subsets:
- addresses:
  - ip: 192.168.1.72
    nodeName: 192.168.1.72
  ports:
  - name: metrics
    port: 9100
    protocol: TCP

on the external node-exporter i have the web.yml with the generated crt and key file and the password generated with htpasswd.

tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key
  # basic_auth_users:
  # prometheus: $2y$10$V2RmZ2wKC7S8jhEz1OXRKOLkq1UHw4qlgpHT.hMg7B447dJQl7RqS

I can use the self-generated certificate by using insecureSkipVerify: true. if i enable basic_auth_users with the user: prometheus and password, it works when i try to access the node-exporter and enter the user/password.

But how to implement the basic_auth into yaml to create the credential / or what is the correct command.? Is there any better way to secure the external node-exporter, if prometheus is deploy with helm?

thanks for your help!

Answer 1

to complete this, below the notes for this topic:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  annotations:
    meta.helm.sh/release-name: prom00
    meta.helm.sh/release-namespace: monitoring-dev
  labels:
    app: node-exporter-vm-agent
    release: prom00
  name: prom00-node-exporter-vm-agent
  namespace: monitoring-dev
spec:
  endpoints:
    - port: metrics
      scheme: https
      basicAuth:
        username:
          key: username
          name: basic-auth
        password:
          key: password
          name: basic-auth
      tlsConfig:
        insecureSkipVerify: true
  jobLabel: jobLabel
  selector:
    matchLabels:
      app: node-exporter-vm-agent
      release: prom00

for the secret: i have create it with: kubectl -n monitoring-dev create secret generic basic-auth --from-literal=username='prometheus' --from-literal=password='password'

on the node-exporter, i have created a web-config.yml file with:

tls_server_config: cert_file: ../cert/prom_node_exp.crt key_file: ../cert/prom_node_expnopass.key basic_auth_users: prometheus: $2y$10$W.nywLSnmQjagtmT6k4uLedGhk1sWMMG3Rspv2r6Z0CzGmLJUveFC

--> the password for the user is created with: htpasswd -nBC 10 "" | tr -d ':\n'