NGINX Ingress Controller Proxying Websocket Connections

Question

I would like to proxy websocket connections dynamically using a snippet.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    nginx.org/path-regex: "case_insensitive"
    nginx.org/proxy-read-timeout: "86400s"
    nginx.org/proxy-send-timeout: "86400s"
    nginx.org/proxy-buffering: "False"
    nginx.org/server-snippets: |
      resolver kube-dns.kube-system.svc.cluster.local valid=30s;
      location ~* "^/ws/(?<username>[^/]+)$" {
          # Prevent malicious characters in the username
          if ($username ~ "[^a-zA-Z0-9_-]") {
              return 400;
          }

          set $service_name svc-$username;
          set $namespace default;

          proxy_http_version 1.1;
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "Upgrade";

          # Forward necessary headers
          proxy_set_header Host $host;
          proxy_set_header X-Forwarded-Host $http_host;
          proxy_set_header X-Forwarded-Proto $scheme;
          proxy_set_header X-Forwarded-For $remote_addr;

          # Upgrade HTTP to WebSocket
          proxy_set_header Upgrade $http_upgrade;
          proxy_set_header Connection "Upgrade";

          # Disable proxy buffering for WebSocket
          proxy_buffering off;

          # Bypass caching
          proxy_cache_bypass $http_upgrade;

          # Pass the WebSocket connection to the user-specific service
          proxy_pass http://$service_name.$namespace.svc.cluster.local;
      }
spec:
  ingressClassName: nginx
  rules:
    - host: "localhost"
      http:
        paths:
          - path: /api/session/start/
            pathType: Prefix
            backend:
              service:
                name: controller-service
                port:
                  number: 8080

It seems like the ingress accepts/upgrades the connection though and websocket messages never make it to the pod.

✗ wscat -c ws://localhost/ws/testuser                                 
Hello
✗ kubectl logs pod/pod-testuser                                       
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
<---- there should be logs here...

# WebSocket endpoint with dynamic username path
@app.websocket("/ws/{username}")
async def websocket_endpoint(websocket: WebSocket, username: str):
    print("Waiting to accept a websocket connection...")
    await websocket.accept()
    print(f"Connected. Waiting for messages from user: {username}")
    while True:
        try:
            message = await websocket.receive_text()
            print(f"Received message from {username}: {message}")
            await websocket.send_text(
                f"Message from user: {username}."
            )
        except WebSocketDisconnect:
            print(f"WebSocket connection closed for user: {username}")
            break

I suspect that the NGINX Controller is Accepting and Upgrading the connection, rather than simply proxying to the pod for it to accept and upgrade?

Answer 1

Here's what I ended up doing.

1. I followed this guide for setting up a static websocket location. Verified it works - I am able to connect with wscat and send messages to the static pod.
2. Then I followed troubleshooting instructions to "print all NGINX configuration files".
3. I noted the generated configuration for the /ws endpoint, and copied it to use as my snippet. It looked like below:

annotations:
    nginx.org/server-snippets: |
      location /ws {
                set $service "svc-testuser";
                proxy_http_version 1.1;
                proxy_set_header Upgrade $http_upgrade;
                proxy_set_header Connection $connection_upgrade;
                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;
                proxy_pass http://svc-testuser.default.svc.cluster.local:8008;
        }