Kube-prometheus stack with loadbalancer for external access to prometheus (and grafana)

Question

I have installed the kube-prometheus stach from here and want to expose prometheus and grafana with a loadbalancer to get access to them from another cluster. To acchieve this i have changed the prometheus-service.yaml by adding a type: LoadBalancer. When i try to access the exposed IP, the server says that the connection has timed out. What should i do to be able to access the prometheus server?

The altered prometheus-service.yaml looks like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.38.0
  name: prometheus-k8s
  namespace: monitoring
spec:
  ports:
  - name: web
    port: 9090
    targetPort: web
  - name: reloader-web
    port: 8080
    targetPort: reloader-web
  type: LoadBalancer
  selector:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
  sessionAffinity: ClientIP

Ideas:

• should I alter the networkpolicy to allow for external access? in that case, how?

Answer 1

I found a way to allow it to be exposed, it was networkpolicy. chekc link from github. One has to add a seperate networkpolicy for prometheus to allow external, as shown here:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: prometheus-allow-external
  namespace: monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
  ingress:
  - ports:
    - port: 9090

The problem is that i tought this was already done under the prometheus-networkPolicy.yaml:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/instance: k8s
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.38.0
  name: prometheus-k8s
  namespace: monitoring
spec:
  egress:
  - {}
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: prometheus
    ports:
    - port: 9090
      protocol: TCP
    - port: 8080
      protocol: TCP
  - from:
    - podSelector:
        matchLabels:
          app.kubernetes.io/name: grafana
    ports:
    - port: 9090
      protocol: TCP
  podSelector:
    matchLabels:
      app.kubernetes.io/component: prometheus
      app.kubernetes.io/instance: k8s
      app.kubernetes.io/name: prometheus
      app.kubernetes.io/part-of: kube-prometheus
  policyTypes:
  - Egress
  - Ingress

Can anybody explain what the difference is?