I am running a 2-node K8s cluster on OVH Bare Metal Servers. I've set up MetalLB and Nginx-Ingress.The 2 servers both have public IPs and are not in the same network segment. I've used one of the IPs as the entrypoint for the LB. The deployments I created 3 nginx containers & services to test the forwarding. When I use host based routing, the endpoints are reachable via the internet, but when I use path based forwarding, only the / path is reachable. For the rest, I get the default backend. My host based Ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-resource-2
spec:
ingressClassName: nginx
rules:
- host: nginx.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-deploy-main
port:
number: 80
- host: blue.nginx.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-deploy-blue
port:
number: 80
- host: green.nginx.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx-deploy-green
port:
number: 80
The path based Ingress resource:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-resource-3
spec:
ingressClassName: nginx
rules:
- host: nginx.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
- path: /blue
pathType: Prefix
backend:
service:
name: nginx-deploy-blue
port:
number: 80
- path: /green
pathType: Prefix
backend:
service:
name: nginx-deploy-green
port:
number: 80
The endpoints are all reachable in both cases
# kubectl describe ing ingress-resource-2
Name: ingress-resource-2
Labels: <none>
Namespace: default
Address:
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
nginx.example.com
/ nginx:80 (192.168.107.4:80)
blue.nginx.example.com
/ nginx-deploy-blue:80 (192.168.164.212:80)
green.nginx.example.com
/ nginx-deploy-green:80 (192.168.164.213:80)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 13m nginx-ingress-controller Configuration for default/ingress-resource-2 was added or updated
# kubectl describe ing ingress-resource-3
Name: ingress-resource-3
Labels: <none>
Namespace: default
Address:
Ingress Class: nginx
Default backend: <default>
Rules:
Host Path Backends
---- ---- --------
nginx.example.com
/ nginx:80 (192.168.107.4:80)
/blue nginx-deploy-blue:80 (192.168.164.212:80)
/green nginx-deploy-green:80 (192.168.164.213:80)
Annotations: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AddedOrUpdated 109s nginx-ingress-controller Configuration for default/ingress-resource-3 was added or updated
Getting the Nginx-Ingress logs:
# kubectl -n nginx-ingress logs pod/nginx-ingress-6947fb84d4-m9gkk
W0803 17:00:48.516628 1 flags.go:273] Ignoring unhandled arguments: []
I0803 17:00:48.516688 1 flags.go:190] Starting NGINX Ingress Controller Version=2.3.0 PlusFlag=false
I0803 17:00:48.516692 1 flags.go:191] Commit=979db22d8065b22fedb410c9b9c5875cf0a6dc66 Date=2022-07-12T08:51:24Z DirtyState=false Arch=linux/amd64 Go=go1.18.3
I0803 17:00:48.527699 1 main.go:210] Kubernetes version: 1.24.3
I0803 17:00:48.531079 1 main.go:326] Using nginx version: nginx/1.23.0
2022/08/03 17:00:48 [notice] 26#26: using the "epoll" event method
2022/08/03 17:00:48 [notice] 26#26: nginx/1.23.0
2022/08/03 17:00:48 [notice] 26#26: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
2022/08/03 17:00:48 [notice] 26#26: OS: Linux 5.15.0-41-generic
2022/08/03 17:00:48 [notice] 26#26: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/08/03 17:00:48 [notice] 26#26: start worker processes
2022/08/03 17:00:48 [notice] 26#26: start worker process 27
2022/08/03 17:00:48 [notice] 26#26: start worker process 28
2022/08/03 17:00:48 [notice] 26#26: start worker process 29
2022/08/03 17:00:48 [notice] 26#26: start worker process 30
2022/08/03 17:00:48 [notice] 26#26: start worker process 31
2022/08/03 17:00:48 [notice] 26#26: start worker process 32
2022/08/03 17:00:48 [notice] 26#26: start worker process 33
2022/08/03 17:00:48 [notice] 26#26: start worker process 34
I0803 17:00:48.543403 1 listener.go:54] Starting Prometheus listener on: :9113/metrics
2022/08/03 17:00:48 [notice] 26#26: start worker process 35
2022/08/03 17:00:48 [notice] 26#26: start worker process 37
I0803 17:00:48.543712 1 leaderelection.go:248] attempting to acquire leader lease nginx-ingress/nginx-ingress-leader-election...
2022/08/03 17:00:48 [notice] 26#26: start worker process 38
...
2022/08/03 17:00:48 [notice] 26#26: start worker process 86
I0803 17:00:48.645253 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated
I0803 17:00:48.645512 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated
I0803 17:00:48.646550 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated
I0803 17:00:48.646629 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated
I0803 17:00:48.646810 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated
I0803 17:00:48.646969 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated
I0803 17:00:48.647259 1 event.go:285] Event(v1.ObjectReference{Kind:"Secret", Namespace:"nginx-ingress", Name:"default-server-secret", UID:"d8271053-2785-408f-b87b-88b9bb9fc488", APIVersion:"v1", ResourceVersion:"1612716", FieldPath:""}): type: 'Normal' reason: 'Updated' the special Secret nginx-ingress/default-server-secret was updated
2022/08/03 17:00:48 [notice] 26#26: signal 1 (SIGHUP) received from 88, reconfiguring
2022/08/03 17:00:48 [notice] 26#26: reconfiguring
2022/08/03 17:00:48 [notice] 26#26: using the "epoll" event method
2022/08/03 17:00:48 [notice] 26#26: start worker processes
2022/08/03 17:00:48 [notice] 26#26: start worker process 89
2022/08/03 17:00:48 [notice] 26#26: start worker process 90
...
2022/08/03 17:00:48 [notice] 26#26: start worker process 136
2022/08/03 17:00:48 [notice] 27#27: gracefully shutting down
2022/08/03 17:00:48 [notice] 27#27: exiting
2022/08/03 17:00:48 [notice] 35#35: gracefully shutting down
2022/08/03 17:00:48 [notice] 31#31: exiting
2022/08/03 17:00:48 [notice] 38#38: gracefully shutting down
2022/08/03 17:00:48 [notice] 32#32: exiting
2022/08/03 17:00:48 [notice] 30#30: exiting
2022/08/03 17:00:48 [notice] 40#40: gracefully shutting down
2022/08/03 17:00:48 [notice] 35#35: exiting
2022/08/03 17:00:48 [notice] 45#45: gracefully shutting down
2022/08/03 17:00:48 [notice] 40#40: exiting
2022/08/03 17:00:48 [notice] 48#48: gracefully shutting down
2022/08/03 17:00:48 [notice] 47#47: exiting
2022/08/03 17:00:48 [notice] 57#57: gracefully shutting down
2022/08/03 17:00:48 [notice] 52#52: exiting
2022/08/03 17:00:48 [notice] 55#55: gracefully shutting down
2022/08/03 17:00:48 [notice] 55#55: exiting
2022/08/03 17:00:48 [notice] 51#51: gracefully shutting down
2022/08/03 17:00:48 [notice] 51#51: exiting
2022/08/03 17:00:48 [notice] 31#31: exit
2022/08/03 17:00:48 [notice] 34#34: gracefully shutting down
2022/08/03 17:00:48 [notice] 34#34: exiting
2022/08/03 17:00:48 [notice] 41#41: exiting
2022/08/03 17:00:48 [notice] 49#49: gracefully shutting down
....
2022/08/03 17:00:48 [notice] 49#49: exiting
2022/08/03 17:00:48 [notice] 57#57: exit
.....
2022/08/03 17:00:48 [notice] 43#43: exit
2022/08/03 17:00:48 [notice] 58#58: gracefully shutting down
2022/08/03 17:00:48 [notice] 38#38: exiting
2022/08/03 17:00:48 [notice] 53#53: gracefully shutting down
2022/08/03 17:00:48 [notice] 48#48: exiting
2022/08/03 17:00:48 [notice] 59#59: gracefully shutting down
2022/08/03 17:00:48 [notice] 58#58: exiting
2022/08/03 17:00:48 [notice] 62#62: gracefully shutting down
2022/08/03 17:00:48 [notice] 60#60: gracefully shutting down
2022/08/03 17:00:48 [notice] 53#53: exiting
2022/08/03 17:00:48 [notice] 61#61: gracefully shutting down
2022/08/03 17:00:48 [notice] 63#63: gracefully shutting down
2022/08/03 17:00:48 [notice] 64#64: gracefully shutting down
2022/08/03 17:00:48 [notice] 59#59: exiting
2022/08/03 17:00:48 [notice] 65#65: gracefully shutting down
2022/08/03 17:00:48 [notice] 62#62: exiting
2022/08/03 17:00:48 [notice] 60#60: exiting
2022/08/03 17:00:48 [notice] 66#66: gracefully shutting down
2022/08/03 17:00:48 [notice] 67#67: gracefully shutting down
2022/08/03 17:00:48 [notice] 63#63: exiting
2022/08/03 17:00:48 [notice] 68#68: gracefully shutting down
2022/08/03 17:00:48 [notice] 64#64: exiting
2022/08/03 17:00:48 [notice] 61#61: exiting
2022/08/03 17:00:48 [notice] 69#69: gracefully shutting down
2022/08/03 17:00:48 [notice] 65#65: exiting
2022/08/03 17:00:48 [notice] 66#66: exiting
2022/08/03 17:00:48 [notice] 71#71: gracefully shutting down
2022/08/03 17:00:48 [notice] 70#70: gracefully shutting down
2022/08/03 17:00:48 [notice] 67#67: exiting
...
2022/08/03 17:00:48 [notice] 65#65: exit
2022/08/03 17:00:48 [notice] 73#73: gracefully shutting down
...
2022/08/03 17:00:48 [notice] 74#74: exiting
2022/08/03 17:00:48 [notice] 83#83: gracefully shutting down
2022/08/03 17:00:48 [notice] 72#72: exiting
2022/08/03 17:00:48 [notice] 77#77: gracefully shutting down
2022/08/03 17:00:48 [notice] 77#77: exiting
2022/08/03 17:00:48 [notice] 77#77: exit
I0803 17:00:48.780547 1 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"nginx-ingress", Name:"nginx-config", UID:"961b1b89-3765-4eb8-9f5f-cfd8212012a8", APIVersion:"v1", ResourceVersion:"1612730", FieldPath:""}): type: 'Normal' reason: 'Updated' Configuration from nginx-ingress/nginx-config was updated
I0803 17:00:48.780573 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"delivery-ingress", UID:"23f93b2d-c3c8-48eb-a2a1-e2ce0453677f", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1527358", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/delivery-ingress was added or updated
I0803 17:00:48.780585 1 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-resource-3", UID:"66ed1c4b-54ae-4880-bf08-49029a93e365", APIVersion:"networking.k8s.io/v1", ResourceVersion:"1622747", FieldPath:""}): type: 'Normal' reason: 'AddedOrUpdated' Configuration for default/ingress-resource-3 was added or updated
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 72
2022/08/03 17:00:48 [notice] 26#26: worker process 72 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 30
2022/08/03 17:00:48 [notice] 26#26: worker process 30 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 35 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 77 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 73
2022/08/03 17:00:48 [notice] 26#26: worker process 73 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 37
2022/08/03 17:00:48 [notice] 26#26: worker process 29 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 32 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 37 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 38 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 41 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 47 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 49 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 63 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 64 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 75 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 47
2022/08/03 17:00:48 [notice] 26#26: worker process 34 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 43 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 48 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 53 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 54 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 59 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 61 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 66 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 55
2022/08/03 17:00:48 [notice] 26#26: worker process 50 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 55 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 83
2022/08/03 17:00:48 [notice] 26#26: worker process 28 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 31 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 42 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 51 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 52 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 56 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 62 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 68 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 71 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 83 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 33
2022/08/03 17:00:48 [notice] 26#26: worker process 33 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 58
2022/08/03 17:00:48 [notice] 26#26: worker process 58 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 57
2022/08/03 17:00:48 [notice] 26#26: worker process 27 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 57 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
2022/08/03 17:00:48 [notice] 26#26: signal 17 (SIGCHLD) received from 40
2022/08/03 17:00:48 [notice] 26#26: worker process 40 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 45 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 60 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 65 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 67 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 69 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 70 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 74 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: worker process 86 exited with code 0
2022/08/03 17:00:48 [notice] 26#26: signal 29 (SIGIO) received
I'm not sure what the issue is, and I can't figure out why it's working when I use different hosts, and not working when I try to use different paths.
I thought it could be resource limits, but I only have the requests, no limits. There is already a default IngressClass
I installed the ingress controller via manifests following the steps here
Update: To add the deployments running in the cluster.
# nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: nginx
name: nginx
spec:
replicas: 1
selector:
matchLabels:
run: nginx-main
template:
metadata:
labels:
run: nginx-main
spec:
containers:
- image: nginx
name: nginx
# nginx-deploy-green.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: nginx
name: nginx-deploy-green
spec:
replicas: 1
selector:
matchLabels:
run: nginx-green
template:
metadata:
labels:
run: nginx-green
spec:
volumes:
- name: webdata
emptyDir: {}
initContainers:
- name: web-content
image: busybox
volumeMounts:
- name: webdata
mountPath: "/webdata"
command: ["/bin/sh", "-c", 'echo "<h1>I am <font color=green>GREEN</font></h1>" > /webdata/index.html']
containers:
- image: nginx
name: nginx
volumeMounts:
- name: webdata
mountPath: "/usr/share/nginx/html"
# nginx-deploy-blue.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
run: nginx
name: nginx-deploy-blue
spec:
replicas: 1
selector:
matchLabels:
run: nginx-blue
template:
metadata:
labels:
run: nginx-blue
spec:
volumes:
- name: webdata
emptyDir: {}
initContainers:
- name: web-content
image: busybox
volumeMounts:
- name: webdata
mountPath: "/webdata"
command: ["/bin/sh", "-c", 'echo "<h1>I am <font color=blue>BLUE</font></h1>" > /webdata/index.html']
containers:
- image: nginx
name: nginx
volumeMounts:
- name: webdata
mountPath: "/usr/share/nginx/html"
I'm not sure if the issue was in how I had deployed the NGINX Ingress Controller, but I tried using the Ingress-nginx controller, and the ingress objects work perfectly well.
I will still try again with the NGINX Ingress Controller just to be sure, but for now the routing is behaving as expected with the other ingress controller.
Thanks for your help guys. I will update if I get it to work with the original Ingress Controller.