Search code examples
kuberneteskubernetes-ingressnginx-ingressmicrok8smetallb

Microk8s + metallb + ingress

Im quite new to kubernetes and Im trying to set up a microk8s test environment on a VPS with CentOS.

What I did:

I set up the cluster, enabled the ingress and metallb

microk8s enable ingress
microk8s enable metallb

Exposed the ingress-controller service:

apiVersion: v1
kind: Service
metadata:
  name: ingress
  namespace: ingress
spec:
  type: LoadBalancer
  selector:
    name: nginx-ingress-microk8s 
  ports:
  - name: http
    protocol: TCP
    port: 80
    targetPort: 80
  - name: https
    protocol: TCP
    port: 443
    targetPort: 443

Exposed an nginx deployment to test the ingress

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    run: nginx
  name: nginx-deploy
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx-deploy
  template:
    metadata:
      labels:
        run: nginx-deploy
    spec:
      containers:
      - image: nginx
        name: nginx

This is the status of my cluster:

NAMESPACE            NAME                                             READY   STATUS    RESTARTS   AGE
kube-system          pod/hostpath-provisioner-5c65fbdb4f-m2xq6        1/1     Running   3          41h
kube-system          pod/coredns-86f78bb79c-7p8bs                     1/1     Running   3          41h
kube-system          pod/calico-node-g4ws4                            1/1     Running   6          42h
kube-system          pod/calico-kube-controllers-847c8c99d-xhmd7      1/1     Running   4          42h
kube-system          pod/metrics-server-8bbfb4bdb-ggvk7               1/1     Running   0          41h
kube-system          pod/kubernetes-dashboard-7ffd448895-ktv8j        1/1     Running   0          41h
kube-system          pod/dashboard-metrics-scraper-6c4568dc68-l4xmg   1/1     Running   0          41h
container-registry   pod/registry-9b57d9df8-xjh8d                     1/1     Running   0          38h
cert-manager         pod/cert-manager-cainjector-5c6cb79446-vv5j2     1/1     Running   0          12h
cert-manager         pod/cert-manager-794657589-srrmr                 1/1     Running   0          12h
cert-manager         pod/cert-manager-webhook-574c9758c9-9dwr6        1/1     Running   0          12h
metallb-system       pod/speaker-9gjng                                1/1     Running   0          97m
metallb-system       pod/controller-559b68bfd8-trk5z                  1/1     Running   0          97m
ingress              pod/nginx-ingress-microk8s-controller-f6cdb      1/1     Running   0          65m
default              pod/nginx-deploy-5797b88878-vgp7x                1/1     Running   0          20m

NAMESPACE            NAME                                TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                      AGE
default              service/kubernetes                  ClusterIP      10.152.183.1     <none>         443/TCP                      42h
kube-system          service/kube-dns                    ClusterIP      10.152.183.10    <none>         53/UDP,53/TCP,9153/TCP       41h
kube-system          service/metrics-server              ClusterIP      10.152.183.243   <none>         443/TCP                      41h
kube-system          service/kubernetes-dashboard        ClusterIP      10.152.183.225   <none>         443/TCP                      41h
kube-system          service/dashboard-metrics-scraper   ClusterIP      10.152.183.109   <none>         8000/TCP                     41h
container-registry   service/registry                    NodePort       10.152.183.44    <none>         5000:32000/TCP               38h
cert-manager         service/cert-manager                ClusterIP      10.152.183.183   <none>         9402/TCP                     12h
cert-manager         service/cert-manager-webhook        ClusterIP      10.152.183.99    <none>         443/TCP                      12h
echoserver           service/echoserver                  ClusterIP      10.152.183.110   <none>         80/TCP                       72m
ingress              service/ingress                     LoadBalancer   10.152.183.4     192.168.0.11   80:32617/TCP,443:31867/TCP   64m
default              service/nginx-deploy                ClusterIP      10.152.183.149   <none>         80/TCP                       19m

NAMESPACE        NAME                                               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR                 AGE
kube-system      daemonset.apps/calico-node                         1         1         1       1            1           kubernetes.io/os=linux        42h
metallb-system   daemonset.apps/speaker                             1         1         1       1            1           beta.kubernetes.io/os=linux   97m
ingress          daemonset.apps/nginx-ingress-microk8s-controller   1         1         1       1            1           <none>                        65m

NAMESPACE            NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
kube-system          deployment.apps/hostpath-provisioner        1/1     1            1           41h
kube-system          deployment.apps/coredns                     1/1     1            1           41h
kube-system          deployment.apps/calico-kube-controllers     1/1     1            1           42h
kube-system          deployment.apps/metrics-server              1/1     1            1           41h
kube-system          deployment.apps/dashboard-metrics-scraper   1/1     1            1           41h
kube-system          deployment.apps/kubernetes-dashboard        1/1     1            1           41h
container-registry   deployment.apps/registry                    1/1     1            1           38h
cert-manager         deployment.apps/cert-manager-cainjector     1/1     1            1           12h
cert-manager         deployment.apps/cert-manager                1/1     1            1           12h
cert-manager         deployment.apps/cert-manager-webhook        1/1     1            1           12h
metallb-system       deployment.apps/controller                  1/1     1            1           97m
default              deployment.apps/nginx-deploy                1/1     1            1           20m

NAMESPACE            NAME                                                   DESIRED   CURRENT   READY   AGE
kube-system          replicaset.apps/hostpath-provisioner-5c65fbdb4f        1         1         1       41h
kube-system          replicaset.apps/coredns-86f78bb79c                     1         1         1       41h
kube-system          replicaset.apps/calico-kube-controllers-847c8c99d      1         1         1       42h
kube-system          replicaset.apps/metrics-server-8bbfb4bdb               1         1         1       41h
kube-system          replicaset.apps/kubernetes-dashboard-7ffd448895        1         1         1       41h
kube-system          replicaset.apps/dashboard-metrics-scraper-6c4568dc68   1         1         1       41h
container-registry   replicaset.apps/registry-9b57d9df8                     1         1         1       38h
cert-manager         replicaset.apps/cert-manager-cainjector-5c6cb79446     1         1         1       12h
cert-manager         replicaset.apps/cert-manager-794657589                 1         1         1       12h
cert-manager         replicaset.apps/cert-manager-webhook-574c9758c9        1         1         1       12h
metallb-system       replicaset.apps/controller-559b68bfd8                  1         1         1       97m
default              replicaset.apps/nginx-deploy-5797b88878                1         1         1       20m

It looks like Metallb works, as the ingress services received an ip from the pool I specified. Now, when I try to deploy an ingress to reach the nginx deployment, I dont get the ADDRESS:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: ingress-nginx-deploy
spec:
  rules:
  - host: test.com
    http:
      paths:
      - backend:
          serviceName: nginx-deploy
          servicePort: 80

NAMESPACE   NAME                   CLASS    HOSTS                       ADDRESS   PORTS   AGE
default     ingress-nginx-deploy   <none>   test.com                              80      13m

An help would be really appreciated. Thank you!

Solution

  • TL;DR

    There are some ways to fix your Ingress so that it would get the IP address.

    You can either:

    Example of Ingress resource that will fix your issue:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-nginx-deploy
spec:
  ingressClassName: public 
  # above field is optional as microk8s default ingressclass will be assigned
  rules:
  - host: test.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-deploy
            port:
              number: 80

    You can read more about IngressClass by following official documentation:

    I've included more explanation that should shed some additional light on this particular setup.

    After you apply above Ingress resource the output of:

    • $ kubectl get ingress

    Will be following:

    NAME                   CLASS    HOSTS      ADDRESS     PORTS   AGE
ingress-nginx-deploy   public   test.com   127.0.0.1   80      43s

    As you can see the ADDRESS contains 127.0.0.1. It's because this particular Ingress controller enabled by an addon, binds to your host (MicroK8S node) to ports 80,443.

    You can see it by running:

    • $ sudo microk8s kubectl get daemonset -n ingress nginx-ingress-microk8s-controller -o yaml

    A side note!

    Look for hostPort and securityContext.capabilities.

    The Service of type LoadBalancer created by you will work with your Ingress controller but it will not be displayed under ADDRESS in $ kubectl get ingress.

    A side note!

    Please remember that in this particular setup you will need to connect to your Ingress controller with a Header Host: test.com unless you have DNS resolution configured to support your setup. Otherwise you will get a 404.

    Additional resource: