Nginx ingress is not resolving path properly for kube-prometheus-stack

Question

I created simple kubernetes cluster with demo app. When creating the cluster I installed prometheus-stack and nginx-ingress-controller with helm (default values files).

After the cluster is set up I create Ingress object to expose prometheus, grafana and alertmanager with:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: monitoring-ingress
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - http:
      paths:
      - path: /prometheus(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: prometheus-operated
            port:
              number: 9090
      - path: /alertmanager(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-kube-prom-alertmanager
            port:
              number: 9093
      - path: /grafana(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-grafana
            port:
              number: 80

When I try to access prometheus via <ingress-controller's external IP>/prometheus it resolves to <ingress-controller's external IP>/graph and displays 404 - page not found.

If I use for example kubectl port-forward svc/prometheus-operated 9090:9090 -n monitoring I can reach prometheus without problem.

I can reach alertmanager through <ingress-controller's external IP>/alertmanager. The path is resolved to <ingress-controller's external IP>/alertmanager/#/alerts

I suspect there is something wrong with the path rewriting but can't figure out what.

Please help...

Answer 1

In the end I was able to find 2 solutions to this problem.

Option 1

I had DNS zone in azure (where my cluster lives as well) and there I added subdomains for grafana, prometheus and alertmanager pointing to the ingress-controller external IP.

When deploying kube-prometheus-stack with Helm Chart I provided default root path for prometheus in values.yaml file with folowing configuration:

prometheus:
  prometheusSpec:
    externalUrl: http://prometheus.mydomainname.something

The Ingress manifest for (for example Prometheus) then needs to include the host address and contain only root path "/".

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-new
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - host: prometheus.mydomainname.something
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-kube-prom-prometheus
            port:
              number: 9090

Option 2

You can also provide default paths for each of the applications from kube-prometheus-stack (prometheus, grafana, alertmanager) as:

prometheus:
  prometheusSpec:
    externalUrl: http://mydomainname.something/prometheus

and make Ingress manifest to redirect based on the path with:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-new
  namespace: monitoring
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - path: /prometheus
        pathType: Prefix
        backend:
          service:
            name: prometheus-stack-kube-prom-prometheus
            port:
              number: 9090