Search code examples
kuberneteslets-encryptportforwardingnginx-ingressk3s

Kubernetes cluster behind ISP router

I'm trying to create a k3s cluster at home using 3 raspberry pi 4. I've sweat a lot setting up the nginx-ingress + letsencrypt pods. Actually, it seems to work but I can't check it now.

Now my problem is the following:

I'm connected to internet using an ISP router (Livebox Orange).

My 3 RPI are connected to it and have the following IPs:

  • Master : 192.168.1.20
  • Node1 : 192.168.1.21
  • Node2 : 192.168.1.22

I've linked my domain name to a dynamic DNS pointing to my ISP box IP (yes this sh*tty box can't handle a persitent IP).

The Load balancer has a range from 192.168.1.240 to 192.168.1.250.

Now my question is how to forward port from the ISP to the load balancer ?

I have a config UI on it which allows me to redirect ports to existing device IP, but as the loadbalancer IP is not a real device, I cannot select it.

I'm sorry I'm a real noob on kubernetes. If you need more details, just ask, I'll provide them.

Thank you in advance.

Solution

  • For your current case:

    You need to introduce a new k3s independent component in your network and that can be a reverse proxy like HAProxy, which can be set up to balance requests between 3 IPs.

    Or:

    #1 Rebuild your k3s cluster without Traefik and Service LoadBalancer:

    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC='server --no-deploy servicelb --no-deploy traefik --flannel-backend=host-gw --write-kubeconfig-mode 644' sh -

    #2 Deploy MetalLB

    https://metallb.universe.tf/configuration/#layer-2-configuration with the configuration:

    ---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.20-192.168.1.22

    This will add a functionality to the cluster where any service of load balancer type will be bound to the IP of the node.

    example from my own:

    NAME     STATUS   ROLES                  AGE     VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
node02   Ready    <none>                 13d     v1.20.2+k3s1   192.168.1.202   <none>        Alpine Linux v3.13   5.10.12-0-virt   containerd://1.4.3-k3s1
node03   Ready    <none>                 2d11h   v1.20.2+k3s1   192.168.1.203   <none>        Alpine Linux v3.13   5.10.12-0-virt   containerd://1.4.3-k3s1
node01   Ready    control-plane,master   13d     v1.20.2+k3s1   192.168.1.201   <none>        Alpine Linux v3.13   5.10.12-0-virt   containerd://1.4.3-k3s1

    NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                                     AGE   SELECTOR
kube-dns                    ClusterIP      10.43.0.10      <none>          53/UDP,53/TCP,9153/TCP                      13d   k8s-app=kube-dns
metrics-server              ClusterIP      10.43.254.20    <none>          443/TCP                                     13d   k8s-app=metrics-server
traefik                     LoadBalancer   10.43.130.1     192.168.1.201   80:31666/TCP,443:31194/TCP,8080:31199/TCP   13d   app=traefik

    #3 follow this setup to get traefik going instead of nginx https://github.com/sleighzy/k3s-traefik-v2-kubernetes-crd the author uses 2 rpis.

    #4 From your router you should forward ports 80/443 to the node which has traffic on. You can get away without an external LoadBalancer because the service traefik in that setup is defined as LoadBalancer and will harbour on one node only. It can be that it migrates between restarts of nodes - you'll have to adjust your router.