EKS Ingress with Single ALB, multiple namespaces, and External DNS

Question

I'm trying to configure a single ALB across multiple namespaces in aws EKS, each namespace has its own ingress resource.

I'm trying to configure the ingress controller aws-loadbalancer-controller on a k8s v1.20.

The problem i'm facing is that each time I try to deploy a new service it always spin-up a new classic loadbalancer in addition to the shared ALB specified in the ingress config.

https://kubernetes-sigs.github.io/aws-load-balancer-controller/v2.2/

# service-realm1-dev.yaml:
apiVersion: v1
kind: Service
metadata:
  name: sentinel
  annotations:
    external-dns.alpha.kubernetes.io/hostname: realm1.dev.sentinel.mysite.io
  namespace: realm1-dev
  labels:
    run: sentinel
spec:
  ports:
    - port: 5001
      name: ps1
      protocol: TCP
  selector:
    app: sentinel
  type: LoadBalancer

# ingress realm1-app
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: sentinel-ingress
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
    alb.ingress.kubernetes.io/success-codes: 200-300
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80}]'
  name: sentinel-ingress-controller
  namespace: realm1-dev
spec:
  rules:
    - host: realm1.dev.sentinel.mysite.io
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              servicePort: use-annotation
              serviceName: sentinel

Also I'm using external dns to create a route53 reecodset, and then I use the same configured DNS to route requests to the specific eks service, is there any issue with this approach ?

Answer 1

I was able to make it work using only one ALB, @YYashwanth, Using Nginx was my fallback plan, I'm trying to make the configuration as simple as possible, maybe in the future when we will try to deploy our solution in other cloud providers we will use nginx ingress controller.

1- To start the service type should be node port, use loadbalancer will create a classic LB.

apiVersion: v1
kind: Service
metadata:
  name: sentinel-srv
  annotations:
    external-dns.alpha.kubernetes.io/hostname: operatorv2.dev.sentinel.mysite.io
  namespace: operatorv2-dev
  labels:
    run: jsflow-sentinel
spec:
  ports:
    - port: 80
      targetPort: 80
      name: ps1
      protocol: TCP
  selector:
    app: sentinel-app
  type: NodePort

2- Second we need to configure group.name, for the ingress controller to merge all ingress configurations using the annotation alb.ingress.kubernetes.io/group.name

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/healthcheck-port: traffic-port
    alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: "5"
    alb.ingress.kubernetes.io/healthy-threshold-count: "2"
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80} ]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/success-codes: "200"
    alb.ingress.kubernetes.io/tags: createdBy=aws-controller
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/unhealthy-threshold-count: "2"
    external-dns.alpha.kubernetes.io/hostname: operatorv2.sentinel.mysite.io
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: sentinel-group
  name: dev-operatorv2-sentinel-ingress-controller
  namespace: operatorv2-dev
spec:
  rules:
    - host: operatorv2.dev.sentinel.mysite.io
      http:
        paths:
          - path: /*
            backend:
              servicePort: 80
              serviceName: sentinel-srv