Search code examples
nginxkuberneteskubernetes-ingressnginx-ingress

Serving HTTP/HTTPS service which is outside of Kubernetes cluster through Ingress

My aim is to route local HTTP service that is not in Kubernetes through Kubernetes Ingress.

The configuration below works so I'm able to open http://owncloud.example.com or https://owncloud.example.com from outside.

Here is Kubernetes configuration:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: owncloud
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/ssl-redirect: "true" 
    ingress.kubernetes.io/secure-backends: "true"    
    ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/server-snippet:
      location ~ ^/(.*) {
        proxy_pass http://192.168.250.100:8260/$1;
        proxy_set_header Host $host;
      }
      location ~ ^/api(.*) {
        proxy_pass http://192.168.250.100:8261/$1;
        proxy_set_header Host $host;
      }
spec:
  tls:
  - hosts:
    - owncloud.example.com
    secretName: owncloud-tls

  rules:
  - host: owncloud.example.com

The issue is that I see some strange errors in browser's Javascript console related to "meta". They are related with deep Javascript code. So unfortunately, there is no useful log. The website produces weird behaviour at few places while locally it works fine. So it seems this is something to do with Kubernetes Ingress.

Previously I used plain Nginx connected to outside and this worked great:

location / {
  proxy_pass http://192.168.250.100:8260/
}

If I add exactly the same block to server-snippet, website doesn't load at all. It catches default Ingress.

How to properly proxy_pass traffic from Kubernetes Ingress to another service which is running outside of Kubernetes? So it doesn't miss something through proxy.

Would be nice to have exploration on server-snippet to understand how Kubernetes Ingress configuration is different from standard Nginx usage.

If using different options, I was not able to find a solution to proxy_pass to different http when accessing path /api.

----------------- Updates -----------------

I have collected all issues for comparison.

Locally - working one:

enter image description here

If I click on manifest.json, it shows "Nothing to preview". If I use wget to download that json, I can see <!DOCTYPE html> in this first line. It's HTML file downloaded. But I can confirm this local version is working perfectly for years. So this screenshot is just to know how it looks when it works.

Through Ingress - not working:

I logged in successfully. Didn't spot anything weird from user experience, but issue exists:

enter image description here

Tried to log out. I'm not able to do it. It throws Owncloud specific error "Access forbidden CSRF check failed" and on the console I see this:

enter image description here

If I go to https://owncloud.example.com/login page on purpose:

enter image description here

If I try to access files on this Owncloud, it also fails with 400:

enter image description here

If I add additional annotations:

  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~ ^/?(.*) {
        proxy_pass http://192.168.250.100:8260/$1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
      }
    # Owncloud tuning
    nginx.ingress.kubernetes.io/proxy-body-size: "500000m"
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "500000m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "36000s"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "36000s"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "36000s"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-redirect-from: "off"
    nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive"

enter image description here

Summarise

No errors on application side. So first thing that comes to my mind is /logout behaviour. I get 412 HTTP code which means: Precondition Failed client error response code indicates that access to the target resource has been denied and 400 bad request error.

Any expertise to catch this issue?

Many thanks

Solution

  • Finally found a working solution.

    I just corrected location and proxy_pass to solve the root cause.

    So if you have some local HTTP service which is outside of Kubernetes cluster and you want to serve this through Ingress, you just need this:

    kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: owncloud
  namespace: default
  annotations:
    kubernetes.io/ingress.class: nginx
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/secure-backends: "true"
    ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~ "^/(.*)" {
        proxy_pass http://192.168.250.100:8260;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto https;

        # Owncloud tuning
        proxy_max_temp_file_size 0;
        client_max_body_size 500000m;
        proxy_read_timeout 36000s;
        proxy_send_timeout 36000s;
        proxy_connect_timeout 36000s;
        proxy_buffering off;
        proxy_redirect off;
        proxy_set_header Connection "Keep-Alive";
      }
    # Owncloud tuning
    nginx.ingress.kubernetes.io/proxy-max-temp-file-size: "0"
    nginx.ingress.kubernetes.io/proxy-body-size: "500000m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "36000s"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "36000s"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "36000s"
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-redirect-from: "off"
    nginx.ingress.kubernetes.io/connection-proxy-header: "keep-alive"
spec:
  rules:
  - host: owncloud.example.com
  tls:
  - hosts:
    - owncloud.example.com
    secretName: owncloud-example-tls

    • Remove Owncloud tuning block if you have another service

    • Remove ssl, secure, X-Forwarded-Proto and tls: bits if you don't need HTTPS

    You can add more location blocks such as ~ "^/api/(.*)" so it works as normal Nginx.

    In my case it was useful to route some local Docker Compose and old fashion services to outside through Kubernetes Ingress.

    P.S. Don't forget to vote for @mWatney comment if you came here to solve Owncloud CSRF error.