Search code examples
kuberneteskubernetes-ingressdocker-desktop

Kubernetes nginx ingress returning 400

What I did:

  1. Install Docker Desktop for Windows
  2. Enabled Kubernetes (throught "Settings" -> "Kubernetes" -> "Enable Kubernetes")
  3. Installed Nginx Ingress via YAML manifest: https://kubernetes.github.io/ingress-nginx/deploy/
  4. Installed kubernetes-dashboard via YAML manifest: https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/#deploying-the-dashboard-ui
  5. Checked that dashboard services and pods working fine: enter image description here
  6. Took a minimal Ingress example (https://kubernetes.io/docs/concepts/services-networking/ingress/#the-ingress-resource) and created ingress manifest on it:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: root-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /dashboard
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443

  1. Ingress was succefuly started. enter image description here

  2. I am keep getting 400 response from that path. enter image description here

  3. curl -k -v https://localhost response on localhost:

PS C:\Windows\system32> curl.exe -k -v https://localhost
    *   Trying [::1]:443...
    * Connected to localhost (::1) port 443
    * schannel: disabled automatic use of client certificate
    * ALPN: curl offers http/1.1
    * ALPN: server accepted http/1.1
    * using HTTP/1.1
    > GET / HTTP/1.1
    > Host: localhost
    > User-Agent: curl/8.4.0
    > Accept: */*
    >
    * schannel: remote party requests renegotiation
    * schannel: renegotiating SSL/TLS connection
    * schannel: SSL/TLS connection renegotiated
    * schannel: remote party requests renegotiation
    * schannel: renegotiating SSL/TLS connection
    * schannel: SSL/TLS connection renegotiated
    < HTTP/1.1 404 Not Found
    < Date: Mon, 22 Jan 2024 20:20:40 GMT
    < Content-Type: text/html
    < Content-Length: 146
    < Connection: keep-alive
    < Strict-Transport-Security: max-age=15724800; includeSubDomains
    <
    <html>
    <head><title>404 Not Found</title></head>
    <body>
    <center><h1>404 Not Found</h1></center>
    <hr><center>nginx</center>
    </body>
    </html>
    * Connection #0 to host localhost left intact
  1. curl -k -v https://localhost response on localhost/dashboard:
    PS C:\Windows\system32> curl.exe -k -v https://localhost/dashboard
    *   Trying [::1]:443...
    * Connected to localhost (::1) port 443
    * schannel: disabled automatic use of client certificate
    * ALPN: curl offers http/1.1
    * ALPN: server accepted http/1.1
    * using HTTP/1.1
    > GET /dashboard HTTP/1.1
    > Host: localhost
    > User-Agent: curl/8.4.0
    > Accept: */*
    >
    * schannel: remote party requests renegotiation
    * schannel: renegotiating SSL/TLS connection
    * schannel: SSL/TLS connection renegotiated
    * schannel: remote party requests renegotiation
    * schannel: renegotiating SSL/TLS connection
    * schannel: SSL/TLS connection renegotiated
    < HTTP/1.1 400 Bad Request
    < Date: Mon, 22 Jan 2024 20:23:04 GMT
    < Transfer-Encoding: chunked
    < Connection: keep-alive
    < Strict-Transport-Security: max-age=15724800; includeSubDomains
    <
    Client sent an HTTP request to an HTTPS server.
    * Connection #0 to host localhost left intact
Solution

  • Quick fix

    Just use NodePort

    apiVersion: v1
kind: Service
metadata:
  name: kubernetes-dashboard-service
  namespace: kubernetes-dashboard
spec:
  selector:
    k8s-app: kubernetes-dashboard
  type: NodePort
  ports:
    - protocol: TCP
      port: 8443
      targetPort: 8443
      nodePort: 31000

    Dashboard will be available at https://localhost:31000

    Root cause

    What you are sending looks like this
    Browser ==HTTPS==> ingress ==HTTP==> k8s-dashboard
    But expected is
    Browser ==HTTPS==> ingress ==HTTPS==> k8s-dashboard

    The issues it that k8s-dashboard service expects HTTPS request, this is the reason why you get Client sent an HTTP request to an HTTPS server.
    It was decrypted by ingress and later on forward by using plain HTTP

    Solution

    1 Export TCP port without using ingress

    Instead of using ingress you can use NodePort (I gave it in Quick fix) or LoadBalancer. Idea is to work on ISO Layer 4(IP) instead of routing it using Layer 7(HTTP)

    2 Use ssl-passthrough from nginx-ingress

    To make it work it needs two changes

    1. Modify nginx ingress controller using kubectl edit
    kubectl -n ingress-nginx edit deployment ingress-nginx-controller

    You have to add - --enable-ssl-passthrough(just one line) to the args
    After change it should look like

    ...
      containers:
      - args:
        - /nginx-ingress-controller
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
        - --election-id=ingress-nginx-leader
        - --controller-class=k8s.io/ingress-nginx
        - --ingress-class=nginx
        - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
        - --validating-webhook=:8443
        - --validating-webhook-certificate=/usr/local/certificates/cert
        - --validating-webhook-key=/usr/local/certificates/key
        - --enable-ssl-passthrough
...
    1. Add proper ingress route

    When route is created you can't specify path like /dashboard because k8s dashboard doesn't know it's working in some subpath, so using rewrite will not resolve it as links generated by dashboard will be wrong(I wrote more about it here)

    So we need ingress route like this (with ssl-passthrough and backend-protocol)

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: root-ingress
  namespace: kubernetes-dashboard
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
spec:
  ingressClassName: nginx
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443

    3 Disable https in kubernetes dashboard (I didn't try)

    When you disable HTTPS in kubernetes dashboard your flow will look like this
    Browser ==HTTPS==> ingress ==HTTP==> k8s-dashboard

    It will perfectly match what you have right now and should work without issues
    As I didn't try it I will leave this article how to do it.
    There is also raw manifest but maybe a little old