How to access the prometheus & grafana via Istion ingress gateway? I have installed the promethius anfd grafana through Helm

Question

I used below command to bring up the pod:

kubectl create deployment grafana --image=docker.io/grafana/grafana:5.4.3 -n monitoring

Then I used below command to create custerIp:

kubectl expose deployment grafana --type=ClusterIP --port=80 --target-port=3000 --protocol=TCP -n monitoring

Then I have used below virtual service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana
spec:
  hosts:
  - "*"
  gateways:
  - cogtiler-gateway.skydeck
  http:
  - match:
    - uri:
        prefix: /grafana
    route:
    - destination:
        port:
          number: 3000
        host: grafana

kubectl apply -f grafana-virtualservice.yaml -n monitoring

Output:

virtualservice.networking.istio.io/grafana created

Now, when I try to access it, I get below error from grafana:

 **If you're seeing this Grafana has failed to load its application files

 1. This could be caused by your reverse proxy settings.

 2. If you host grafana under subpath make sure your grafana.ini root_path setting includes subpath

 3. If you have a local dev build make sure you build frontend using: npm run dev, npm run watch, or npm run build

 4. Sometimes restarting grafana-server can help **

Answer 1

The easiest and working out of the box solution to configure that would be with a grafana host and / prefix.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
  namespace: monitoring
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http-grafana
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
  namespace: monitoring
spec:
  hosts:
  - "grafana.example.com"
  gateways:
  - grafana-gateway
  http:
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: grafana
        port:
          number: 80

---

As you mentioned in the comments, I want to use path based routing something like my.com/grafana, that's also possible to configure. You can use istio rewrite to configure that.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
  namespace: monitoring
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http-grafana
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
  namespace: monitoring
spec:
  hosts:
  - "*"
  gateways:
  - grafana-gateway
  http:
  - match:
    - uri:
        prefix: /grafana
    rewrite:
      uri: /
    route:
    - destination:
        host: grafana
        port:
          number: 80

But, according to this github issue you would have also additionally configure grafana for that. As without the proper grafana configuration that won't work correctly.

---

I found a way to configure grafana with different url with the following env variable GF_SERVER_ROOT_URL in grafana deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: grafana
  name: grafana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: grafana
    spec:
      containers:
      - image: docker.io/grafana/grafana:5.4.3
        name: grafana
        env:
        - name: GF_SERVER_ROOT_URL
          value: "%(protocol)s://%(domain)s/grafana/"
        resources: {}

Also there is a Virtual Service and Gateway for that deployment.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: grafana-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http-grafana
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: grafana-vs
spec:
  hosts:
  - "*"
  gateways:
  - grafana-gateway
  http:
  - match:
    - uri:
        prefix: /grafana/
    rewrite:
      uri: /
    route:
    - destination:
        host: grafana
        port:
          number: 80