Search code examples
kubernetestraefikk3s

Services being exposed (traefik-ingress) on multiple IPs, on baremetal

I'm learning Kubernetes at the same time I want to use it for personal (but public) projects. Being personal projects my budget isn't big so I have the following configuration:

  • 1 master node VPS with a public IP 45.77.114.72, and private IP 10.39.96.5.
  • 1 worker node VPS with a public IP 149.28.111.67, and private IP 10.39.96.4.
  • 1 worker node VPS with a public IP 45.77.113.193, and private IP 10.39.96.3.

I've used k3s as the Kubernetes distribution, meaning I have the traefik ingress controller, not nginx. After some testing I was able to setup applications and access them from my domain name using an ingress resource, so no problems there. However I see that my services, as well as the traefik loadbalancer, indicate that all public IPs are being used. This is undesirable since I would much prefer not to expose every node.

I tried adding a firewall (from the service provider) to restrict access on those nodes to only packets coming from the sub-network (so that I can still SSH into them from the master node). This works but the ingress still shows all IPs, and I assume this may actually cause problems if it thinks all are publically reachable.

I also tried setting up a NAT on the master node and disabling the public IPs on the worker nodes, so all traffic goes trough the master node. This worked to remove the worker nodes public IPs from the ingress but now instead their private IP (not even the gateway) is shown.

2robin@pop-os:~$ k get ingress
NAME         CLASS    HOSTS              ADDRESS                              PORTS     AGE
test-nginx   <none>   test.lycraft.net   10.39.96.3,10.39.96.4,45.77.114.72   80, 443   11m

I'm at a lost on what to do.

Solution

  • The addresses listed in an Ingress status are related to your Ingress Controllers - and how to reach them/where they run. This is not specific to Traefik: we would see the same with Nginx or other implementations.

    The reason you have 3 addresses is that you have 3 replicas of Traefik.

    If we want only one of our node to serve as in Ingress for our cluster: in case of a Deployment/ReplicaSet, we would lower the amount of replicas and set a nodeSelector. Or here, with a DaemonSet, just set a nodeSelector, forcing Traefik to start on our designated ingress node.