Kubernetes Ingress Routing HTTP with Kestrel

I'm new to k8s and slowly picking it up. I have built out a web api that runs on .net 5 and uses HTTPS (a previous version used http) only, I built the image in docker compose and everything works as expected locally with the default aspnetapp.pfx cert. What I am struggling with is that my ingress routing seems to terminaate the connection early.

I have created a cert pfx for kestrel to run on with the CN name of a.b.com this was from the crt and key files that are needed to create secrets in the documentation. but from my understanding kestrel needs a pfx to run (straight out of the box).

Below are my ingress, service and deployment snippets as well as the entries from the logs: I believe that my issue is that in the logs it is showing as a "http" request but it should be https

Logs:

2021/02/24 09:44:11 [error] 3231#3231: *44168360 upstream prematurely closed connection while reading response header from upstream, client: 127.0.0.1, server: _, request: "GET /dayofweek/api/1/dayofweek/action HTTP/2.0", upstream: "http://<podip>/api/1/dayofweek/action", host: "<clusterip>:<nodePort>"

Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dayofweek-ingress-path
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    #kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: a.b.com
  -  http:
      paths:
      - backend:
          service:
            name: dayofweek-svc
            port:
              number: 9057
        path: /dayofweek/?(.*)
        pathType: Prefix

Service + Deployment

apiVersion: v1
kind: Service
metadata:
  name: dayofweek-svc
  labels:
    run: dayofweek-svc
spec:
  ports:
  - port: 9057
    targetPort: 3441
    protocol: TCP
    name: https
  selector:
    app: dayofweek
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dayofweek
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dayofweek
  template:
    metadata:
      labels:
        app: dayofweek
    spec:
      volumes:
      - name: cert-volume
        persistentVolumeClaim:
          claimName: persistentcerts-claim
      containers:
      - name: dayofweek
        image: ...omitted
        ports:
        - containerPort: 3441
        env:
          - name: DOTNET_ENVIRONMENT
            value: Development
          - name: Culture
            value: en-US #English by default
          - name: ASPNETCORE_Kestrel__Certificates__Default__Path
            value: /https/aspnetapp.pfx
          - name: ASPNETCORE_Kestrel__Certificates__Default__Password
            value: password
        volumeMounts:
        - mountPath: /https
          name: cert-volume

I followed through with the guide here: https://kubernetes.io/docs/concepts/services-networking/connect-applications-service/

And I seem to have it up and running, but for some reason when I'm not sure if I've overcomplicated it by adding in the "-host" element of the ingress.

Any help would be greatly appreciated!

Solution

Service and deployment look correct, but I can see some issues with ingress.

When using ssl-passthrough path based routing doesn't work so you can skip it.

Also, there is a typo in your config:

- host: a.b.com
-  http:    # <- HERE

there shouldn't be the second dash.

Here is how it should look like:

spec:
  rules:
  - host: a.b.com
    http:
      paths:

Additionally, have a look what nginx ingres docs has to say about ssl-passthrough:

SSL Passthrough

The --enable-ssl-passthrough flag enables the SSL Passthrough feature, which is disabled by default. This is required to enable passthrough backends in Ingress objects.

Warning

This feature is implemented by intercepting all traffic on the configured HTTPS port (default: 443) and handing it over to a local TCP proxy. This bypasses NGINX completely and introduces a non-negligible performance penalty.

SSL Passthrough leverages SNI and reads the virtual domain from the TLS negotiation, which requires compatible clients. After a connection has been accepted by the TLS listener, it is handled by the controller itself and piped back and forth between the backend and the client.

If there is no hostname matching the requested host name, the request is handed over to NGINX on the configured passthrough proxy port (default: 442), which proxies the request to the default backend.

There is also this in docs:

SSL Passthrough

nginx.ingress.kubernetes.io/ssl-passthrough instructs the controller to send TLS connections directly to the backend instead of letting NGINX decrypt the communication. See also TLS/HTTPS in the User guide.

**Note SSL Passthrough is disabled by default and requires starting the controller with the --enable-ssl-passthrough flag.

Attention

Because SSL Passthrough works on layer 4 of the OSI model (TCP) and not on the layer 7 (HTTP), using SSL Passthrough invalidates all the other annotations set on an Ingress object.

So, according to the docs, in order for it to work you need to enable ssl-passthrough feature first. After this is done, you can use ssl-passthrough annotation but this invalidates all the other annotations and path based routing stops working.