K3s traefik ingress returns gateway timeout

Question

I am currently playing around with a rpi based k3s cluster and I am observing a weird phenomenon.

I deployed two applications. The first one is nginx which I can reach on the url http://external-ip/foo based on the following ingress rule:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: foo
  namespace: foo
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
    traefik.ingress.kubernetes.io/rewrite-target: "/"
spec:
  rules:
  - http:
      paths:
      - path: /foo
        backend:
          serviceName: foo-service
          servicePort: 8081

And the other one is grafana which I cannot reach on the url http://external-ip/grafana based on the below ingress rule:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: grafana
  namespace: grafana
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/rule-type: "PathPrefixStrip"
    traefik.ingress.kubernetes.io/rewrite-target: "/"
spec:
  rules:
  - http:
      paths:
      - path: /grafana
        backend: 
          serviceName: grafana-service
          servicePort: 3000

When I do a port-forward directly on the pod I can reach the grafana app, when I use the port-forward on the grafana service it also works.

However as soon as I try to reach it through the subpath I will get a gateway timeout.

Does anyone have a guess what I am missing?

Here the deployment and service for the grafana deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: grafana
  labels:
    app: grafana
    tier: frontend
    service: monitoring
spec:
  selector:
    matchLabels:
      app: grafana
      tier: frontend
  template:
    metadata:
      labels:
        app: grafana
        tier: frontend
        service: monitoring
    spec:
      containers:
      - image: grafana
        imagePullPolicy: IfNotPresent
        name: grafana
        envFrom:
        - configMapRef:
            name: grafana-config
        ports:
        - name: frontend
          containerPort: 3000
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: grafana-service
  namespace: grafana
  labels:
    app: grafana
    tier: frontend
    service: monitoring
spec:
  selector:
    app: grafana
    tier: frontend
  type: NodePort
  ports:
  - name: frontend
    port: 3000
    protocol: TCP
    targetPort: 3000

Solution

I had to add the following two parameters to my configmap to make it work:

GF_SERVER_ROOT_URL=http://localhost:3000/grafana/   
GF_SERVER_FROM_SUB_PATH=true

Answer 1

As I mentioned in comments grafana is not listening on / like default nginx.

There is related github issue about this, and if you want to make it work you should specify root_url

grafana.ini:
  server:
    root_url: https://subdomain.example.com/grafana

Specifically take a look at this and this comment.

---

@tehemaroo add his own solution which include changing root url and sub_path in configmap

I had to add the following two parameters to my configmap to make it work:

GF_SERVER_ROOT_URL=http://localhost:3000/grafana/   
GF_SERVER_FROM_SUB_PATH=true

And related documentation about that

To serve Grafana behind a sub path:

Include the sub path at the end of the root_url.

Set serve_from_sub_path to true.

[server]
domain = example.com
root_url = %(protocol)s://%(domain)s:%(http_port)s/grafana/
serve_from_sub_path = true