Kubernetes:Traefik (v2.2) dashboard returning "404 page not found"

Question

Having problem bringing up Traefik dashboard on Kubernetes environment, the following is my traefik deployment setting:

kind: Deployment
apiVersion: apps/v1
metadata:
  namespace: ingress-traefik
  name: traefik
  labels:
    app: traefik

spec:
  replicas: 1
  selector:
    matchLabels:
      app: traefik
  template:
    metadata:
      labels:
        app: traefik
    spec:
      serviceAccountName: traefik-ingress-controller
      containers:
        - name: traefik
          image: traefik:v2.2
          ports:
            - name: web
              containerPort: 80
            - name: websecure
              containerPort: 443
            - name: admin
              containerPort: 8080
          args:
            - --api
            - --api.dashboard=true
            - --providers.kubernetesingress
            - --providers.kubernetescrd
            - --entrypoints.web.Address=:80
            - --entrypoints.websecure.Address=:443
            - --ping.entryPoint=web
            - --log.level=debug

and the dashboard ingressRoute setting:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: ingress-traefik
spec:
  entryPoints:
    - web
    - websecure
  routes:
  - match: Host(`traefik.example.io`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))
    kind: Rule
    services:
    - name: api@internal
      kind: TraefikService
  tls:
    secretName: cert-stage-wildcard
    domains:
      - main: example.io
        sans:
          - "*.example.io"

When curl the dashboard site returns a "404":

$ curl -v https://traefik.example.io/dashboard
*   Trying 159.203.52.215:443...
* TCP_NODELAY set
* Connected to traefik.example.io (159.203.52.215) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /home/ken.tsoi/anaconda3/ssl/cacert.pem
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.example.io
*  start date: Oct 28 16:10:10 2020 GMT
*  expire date: Jan 26 16:10:10 2021 GMT
*  subjectAltName: host "traefik.example.io" matched cert's "*.example.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /dashboard HTTP/1.1
> Host: traefik.example.io
> User-Agent: curl/7.68.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 29 Oct 2020 17:51:07 GMT
< Content-Length: 19
< 
404 page not found
* Connection #0 to host traefik.example.io left intact

Checking the log of traefik, did not see any updates when doing "curl", however, doing a "grep dashboard" did see the dashboard ingressRoute has been configured:

time="2020-10-29T17:26:53Z" level=debug msg="Configuration received from provider kubernetescrd: {\"http\":{\"routers\":{\"ingress-traefik-traefik-dashboard-e1cd6df5083c9bc8083c\":{\"entryPoints\":[\"web\",\"websecure\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.example.io`) \\u0026\\u0026 (PathPrefix(`/api`) || PathPrefix(`/dashboard`))\",\"tls\":{\"domains\":[{\"main\":\"example.io\",\"sans\":[\"*.example.io\"]}]}}},\"middlewares\":{\"ingress-traefik-traefikbasicauth\":{\"basicAuth\":{\"users\":[\"testuser:$apr1$sS2w2/sx$aw4f8LNSyypdknEUOqcIp/\"],\"realm\":\"Traefik\"}}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=kubernetescrd
time="2020-10-29T17:26:53Z" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=web routerName=ingress-traefik-traefik-dashboard-e1cd6df5083c9bc8083c@kubernetescrd middlewareName=tracing 

(BTW: I have used basicAuth for authentication, it is working so I omit it from the ingressRoute)

Can anyone point out what the problem may be?

Answer 1

This problem had been resolved (Not sure if this is a solution or only a workaround):

Please check here: https://stackoverflow.com/a/64709491/6716204

The key point is, I set api.insecure=true, so that I can bring up the dashboard locally, and then do the routing like other services.