Kubernetes (AKS) Multiple Nginx Ingress Controllers

Question

I have a cluster set up in AKS, and need to install a second Nginx Ingress controller so we can manage the source IP using Local instead of Cluster IP.

Reference:

https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-typeloadbalancer

I know how to do the above, but I am confused on how I can deploy my Services that need to be running on that Ingress Controller from my manifests. I use helm to schedule my deployments. Below is an example of how I am installing the ingress controller currently with Ansible in our pipeline:

  - name: Create NGINX Load Balancer (Internal) w/ IP {{ aks_load_balancer_ip }}
    command: >
      helm upgrade platform ingress-nginx/ingress-nginx -n kube-system
      --set controller.service.loadBalancerIP="{{ aks_load_balancer_ip }}"
      --set controller.service.annotations."service\.beta\.kubernetes\.io\/azure-load-balancer-internal"="true"
      --set controller.replicaCount="{{ nginx_replica_count }}"
      --version {{ nginx_chart_ver }}
      --install

I am aware here, that I will need to set some values to change the metadata name etc since we are using charts from bitnami.

Below is an example of my Ingress charts:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ template "service.fullname" . }}
  labels:
    app: {{ template "service.name" . }}
    chart: {{ template "service.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: {{ .Values.ingress.proxy_body_size }}
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/add-base-url: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "{{ .Values.ingress.proxy_read_timeout }}"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "{{ .Values.ingress.proxy_send_timeout }}"
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "{{ .Values.ingress.proxy_connect_timeout }}"
spec:
  tls:
    - hosts:
        - "{{ .Values.service_url }}"
      secretName: tls-secret
  rules:
  - host: "{{ .Values.service_url }}"
    http:
      paths:
      - pathType: Prefix
        path: "/?(.*)"
        backend:
          service:
            name: {{ template "service.name" . }}
            port:
              number: {{ .Values.service.port }}

I am unsure if there is an annotation that should be applied in the ingress to define which ingress controller this gets assigned to.

I was reading this: https://kubernetes.io/docs/reference/kubernetes-api/service-resources/ingress-v1/#IngressSpec

And feel like I just need to add that underneath spec and define the name of the load balancer? Any assistance would be appreciated.

Answer 1

Solution:

I was able to figure this out myself, I created both my ingresses using the following logic:

      helm install nginx-two ingress-nginx/ingress-nginx -n kube-system \
      --set controller.replicaCount=2 \
      --set controller.service.loadBalancerIP="10.24.1.253" \
      --set controller.ingressClassResource.name=nginx-two \
      --set controller.ingressClass="nginx-two" \
      --set controller.service.annotations."service\.beta\.kubernetes\.io\/azure-load-balancer-internal"="true" \
      --version {{ nginx_chart_ver }}
      --install

      helm install nginx-oneingress-nginx/ingress-nginx -n kube-system \
      --set controller.replicaCount=2 \
      --set controller.service.loadBalancerIP="10.24.1.254" \
      --set controller.ingressClassResource.name=nginx-one \
      --set controller.ingressClass="nginx-one" \
      --set controller.service.annotations."service\.beta\.kubernetes\.io\/azure-load-balancer-internal"="true" \
      --version {{ nginx_chart_ver }}
      --install

Then I was able to reference it in my ingress chart by updating the annotation to the name. I am sure this can be smoothed out further by defining the namespace if I wanted the ingress controller there, but I would need to update my PSP first.

Working Ingress example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ template "service.fullname" . }}
  labels:
    app: {{ template "service.name" . }}
    chart: {{ template "service.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
  annotations:
    kubernetes.io/ingress.class: "nginx-two"