EKS: how to register dynamic generated eks loadbalancer dns in eks cloudmap?

Question

I have one kafka external service in which I have used type: Loadbalancer

Problem: This service will always create new load balancer after uninstall/delete.

Usecase: I want to register load balancer's dns against one static DNS in aws cloud-map.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: kafka-test-3-1-external
  labels:
    helm.sh/chart: kafka-0.21.5
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: kafka-broker
    app.kubernetes.io/name: kafka
    app.kubernetes.io/instance: kafka-test-3
    pod: "kafka-test-3-1"
spec:
  type: LoadBalancer
  ports:
    - name: external-broker
      port: 19092
      targetPort: 19092
      protocol: TCP
#
  selector:
    app.kubernetes.io/component: kafka-broker
    app.kubernetes.io/name: kafka
    app.kubernetes.io/instance: kafka-test-3
    statefulset.kubernetes.io/pod-name: "kafka-test-3-1"

How can I do that?

Answer 1

You can use the below annotations in your services

external-dns.alpha.kubernetes.io/hostname: "kafka-test-3-1.kafka.internal"
external-dns.alpha.kubernetes.io/ttl: "60"

where kafka.internal is my cloud-map namespace.

So service would be look like below snippet.

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: "kafka-test-3-1.kafka.internal"
    external-dns.alpha.kubernetes.io/ttl: "60"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
  name: kafka-test-3-1-external
  labels:
    helm.sh/chart: kafka-0.21.5
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: kafka-broker
    app.kubernetes.io/name: kafka
    app.kubernetes.io/instance: kafka-test-3
    pod: "kafka-test-3-1"
spec:
  type: LoadBalancer
  ports:
    - name: external-broker
      port: 19092
      targetPort: 19092
      protocol: TCP
#
  selector:
    app.kubernetes.io/component: kafka-broker
    app.kubernetes.io/name: kafka
    app.kubernetes.io/instance: kafka-test-3
    statefulset.kubernetes.io/pod-name: "kafka-test-3-1"

To register the service loadbalancer's dns to cloud-map, we need to use external-dns service.

Note: You have to create namespaces in cloudmap. And provide enough access to your kubernetes user.

To use the AWS Cloud Map API, a user must have permissions to create the DNS namespace. Additionally you need to make sure that your nodes (on which External DNS runs) have an IAM instance profile with the AWSCloudMapFullAccess managed policy attached, that provides following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetHostedZone",
        "route53:ListHostedZonesByName",
        "route53:CreateHostedZone",
        "route53:DeleteHostedZone",
        "route53:ChangeResourceRecordSets",
        "route53:CreateHealthCheck",
        "route53:GetHealthCheck",
        "route53:DeleteHealthCheck",
        "route53:UpdateHealthCheck",
        "ec2:DescribeVpcs",
        "ec2:DescribeRegions",
        "servicediscovery:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}

Before applying the below snippet, replace placeholders

• YOUR_NAMESPACE: with your kubernetes' cluster namespace like default
• AWS_REGION_VALUE: ap-south-1
• DOMAIN_FILTER: it must be your cloudmap namespaces' name. In my case it would be kafka.internal

Snippet: external-dns

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: YOUR_NAMESPACE
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.7.6
        env:
          - name: AWS_REGION
            value: AWS_REGION_VALUE # ap-south-1 put your CloudMap NameSpace region
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=DOMAIN_FILTER # Makes ExternalDNS see only the namespaces that match the specified domain. Omit the filter if you want to process all available namespaces.
        - --provider=aws-sd
        - --aws-zone-type=private # Only look at public namespaces. Valid values are public, private, or no value for both)
        - --txt-owner-id=kafka-identifier

External-dns pod will polling for changes in kubernetes services and if find any changes it will use external-dns.alpha.kubernetes.io/hostname annotation to map that loadbalancer's dns into cloud-map namespaces.

For more details for auto register, https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws-sd.md