Search code examples
kubernetesnginxnginx-ingressminio

Cannot expose min.io console via nginx-ingress and Helm chart installation

I am using min.io Operator and Tenant with Helm Chart and ArgoCD (on GCP GKE)

I am having trouble with the Min.io Tenant Console and nginx-ingress (port-forwarding works but I need it exposed to the public)

All requests from browser (or any client) result in HTTP 400. nginx logs are always the same:

2024/01/11 16:38:55 [error] 2469#2469: *16521012 recv() failed (104: Connection reset by peer) while reading upstream, client: XXX.XXX.XXX.XXX, server: XXX, request: "GET / HTTP/2.0", upstream: "[http://172.28.1.49:9443/"](http://172.28.1.49:9443/%22), host: "XXX"

Here is a verbose curl output for https://xxx.xxx/login:

user ~ $ curl GET --verbose https://xxx.xxx/login
* Could not resolve host: GET
* Closing connection 0
curl: (6) Could not resolve host: GET
*   Trying LB-IP:443...
* Connected to xxx.xxx (LB-IP) port 443 (#1)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=xxx.xxx
*  start date: Jan  9 16:32:45 2024 GMT
*  expire date: Apr  8 16:32:44 2024 GMT
*  subjectAltName: host "xxx.xxx" matched cert's "xxx.xxx"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* using HTTP/2
* h2 [:method: GET]
* h2 [:scheme: https]
* h2 [:authority: xxx.xxx]
* h2 [:path: /login]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* Using Stream ID: 1 (easy handle 0x133008200)
> GET /login HTTP/2
> Host: xxx.xxx
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/2 400
< date: Fri, 12 Jan 2024 08:31:20 GMT
< strict-transport-security: max-age=15724800; includeSubDomains
<
Client sent an HTTP request to an HTTPS server.
* Connection #1 to host xxx.xxx left intact

This goes for both the API and the Console (separate ingress definitions, same error)

This upstream IP value is an IP of a pod from the StatefulSet, which at least is not random. No logs in the sts though.

The relevant parts of the ArgoCD app yaml and Helm chart overrides:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: XXX
  namespace: argocd
  annotations:
    # ...
spec:
  destination:
    namespace: minio-tenant
    server: https://kubernetes.default.svc
  project: default
  source:
    repoURL: https://operator.min.io
    chart: tenant
    targetRevision: "5.0.11"
    helm:
      version: v3
      valuesObject:
        tenant:
          name: XXX
          pools:
            - servers: 4
          exposeServices:
            minio: false
            console: false
        ingress:
          api:
            enabled: true
            ingressClassName: "nginx"
            annotations:
              nginx.ingress.kubernetes.io/ssl-redirect: "true"
              nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
              cert-manager.io/cluster-issuer: "letsencrypt-prod"
            tls:
            - hosts:
              - XXXXXX
              secretName: XXX
            host: XXXXXX
            path: /
            pathType: Prefix
          console:
            enabled: true
            ingressClassName: "nginx"
            annotations:
              nginx.ingress.kubernetes.io/ssl-redirect: "true"
              nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
              cert-manager.io/cluster-issuer: "letsencrypt-prod"
            tls:
            - hosts:
              - XXXXXX
              secretName: XXX
            host: XXXXXX
            path: /
            pathType: Prefix

Also tried adding the nginx annotations I have seen in some nginx examples with the same outcome:

nginx.ingress.kubernetes.io/configuration-snippet: |
  proxy_set_header Host $http_host;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_set_header X-NginX-Proxy true;
  real_ip_header X-Real-IP;
  chunked_transfer_encoding off;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
Solution

  • In the end it turned out the minio tenant server was very sensitive to the HTTPS connections. After many many trials & error a rather simple solution made it work.

    1. A correct nginx.ingress.kubernetes.io/upstream-vhost makes sure traffic is routed correctly to the pods
    2. Specifying nginx.ingress.kubernetes.io/backend-protocol instead of the upgrade traffic annotations and headers makes sure only https traffic is sent (which minio console seems to insist on even w. port-forward)
            ingress:
          api:
            enabled: true
            annotations:
              kubernetes.io/ingress.class: nginx
              cert-manager.io/cluster-issuer: "letsencrypt-prod"
              nginx.ingress.kubernetes.io/upstream-vhost: minio
              nginx.ingress.kubernetes.io/backend-protocol: https
            tls:
            - hosts:
              - domain-1
              secretName: secret-1
            host: domain-1
            path: /
            pathType: Prefix
          console:
            enabled: true
            annotations:
              kubernetes.io/ingress.class: nginx
              cert-manager.io/cluster-issuer: "letsencrypt-prod"
              nginx.ingress.kubernetes.io/upstream-vhost: khc-minio-tenant-console:9443
              nginx.ingress.kubernetes.io/backend-protocol: https
            tls:
            - hosts:
              - domain-2
              secretName: secret-2
            host: domain-2
            path: /
            pathType: Prefix