I originally posted this question as an issue on the GitHub project for the AWS Load Balancer Controller here: https://github.com/kubernetes-sigs/aws-load-balancer-controller/issues/2069.
I'm seeing some odd behavior that I can't trace or explain when trying to get the loadBalacnerDnsName from an ALB created by the controller. I'm using v2.2.0 of the AWS Load Balancer Controller in a CDK project. The ingress that I deploy triggers the provisioning of an ALB, and that ALB can connect to my K8s workloads running in EKS.
Here's my problem: I'm trying to automate the creation of a Route53 A Record that points to the loadBalancerDnsName of the load balancer, but the loadBalancerDnsName that I get in my CDK script is not the same as the loadBalancerDnsName that shows up in the AWS console once my stack has finished deploying. The value in the console is correct and I can get a response from that URL. My CDK script outputs the value of the DnsName as a CfnOutput value, but that URL does not point to anything.
In CDK, I have tried to use KubernetesObjectValue to get the DNS name from the load balancer. This isn't working (see this related issue: https://github.com/aws/aws-cdk/issues/14933), so I'm trying to lookup the Load Balancer with CDK's .fromLookup and using a tag that I added through my ingress annotation:
const alb = elbv2.ApplicationLoadBalancer.fromLookup(this, 'appAlb', {
loadBalancerTags: {
Environment: 'test',
},
});
Here's the project in which I'm running into this issue: https://github.com/briancaffey/django-cdk
Here are some relevant files:
Installing the AWS Load Balancer Controller with CDK: https://github.com/briancaffey/django-cdk/blob/main/src/eks/awslbc/index.ts
Ingress object definition: https://github.com/briancaffey/django-cdk/tree/main/src/eks/resources/ingress
ALB fromLookup method that is generating the wrong DnsName:
https://github.com/briancaffey/django-cdk/blob/main/src/django-eks.ts#L297
Does anyone have any ideas about what could be causing this or how I can debug? If there is any other information that would be helpful to collect in my debugging process, please let me know.
I'm not sure if this is possible, but it seems like the Load Balancer might be created, assigned a DnsName, and then maybe fails provisioning at some point and tries to create another ALB that with another DnsName that is successful.
Update: Here are logs from the AWS Load Balancer Controller deployment:
~/git/github/django-cdk$ kubectl logs deployment/goeksstackdjangoekssamplealbingresscontroller48a16415-aws-load -n kube-system
Found 2 pods, using pod/goeksstackdjangoekssamplealbingresscontroller48a16415-aws-c6d8h
{"level":"info","ts":1623379830.094481,"msg":"version","GitVersion":"v2.2.0","GitCommit":"68c417a7ea37ff153f053d9ffef1cc5c70d7e211","BuildDate":"2021-05-14T21:49:05+0000"}
{"level":"info","ts":1623379830.1275837,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1623379830.1310954,"logger":"setup","msg":"adding health check for controller"}
{"level":"info","ts":1623379830.1312613,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-v1-pod"}
{"level":"info","ts":1623379830.131368,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379830.131444,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379830.1315207,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-networking-v1beta1-ingress"}
{"level":"info","ts":1623379830.1316643,"logger":"setup","msg":"starting podInfo repo"}
I0611 02:50:32.131767 1 leaderelection.go:242] attempting to acquire leader lease kube-system/aws-load-balancer-controller-leader...
{"level":"info","ts":1623379832.1318467,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
I0611 02:50:32.150030 1 leaderelection.go:252] successfully acquired lease kube-system/aws-load-balancer-controller-leader
{"level":"info","ts":1623379832.2321155,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1623379832.2321231,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2321854,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2322075,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2323742,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc00053a820"}
{"level":"info","ts":1623379832.2324347,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc00053a870"}
{"level":"info","ts":1623379832.2324626,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2324827,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2325017,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.232624,"logger":"controller","msg":"Starting EventSource","controller":"service","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.2326622,"logger":"controller","msg":"Starting Controller","controller":"service"}
{"level":"info","ts":1623379832.232741,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1623379832.2328286,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1623379832.2332337,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
{"level":"info","ts":1623379832.3324778,"logger":"controller","msg":"Starting EventSource","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.3327508,"logger":"controller","msg":"Starting workers","controller":"service","worker count":3}
{"level":"info","ts":1623379832.332836,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"channel source: 0xc00053a8c0"}
{"level":"info","ts":1623379832.3328674,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.4330835,"logger":"controller","msg":"Starting Controller","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding"}
{"level":"info","ts":1623379832.4333706,"logger":"controller","msg":"Starting workers","reconcilerGroup":"elbv2.k8s.aws","reconcilerKind":"TargetGroupBinding","controller":"targetGroupBinding","worker count":3}
{"level":"info","ts":1623379832.433311,"logger":"controller","msg":"Starting EventSource","controller":"ingress","source":"kind source: /, Kind="}
{"level":"info","ts":1623379832.4334028,"logger":"controller","msg":"Starting Controller","controller":"ingress"}
{"level":"info","ts":1623379832.433451,"logger":"controller","msg":"Starting workers","controller":"ingress","worker count":3}
{"level":"info","ts":1623379851.8822243,"logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"app/app-ingress\",\"resources\":{\"AWS::EC2::SecurityGroup\":{\"ManagedLBSecurityGroup\":{\"spec\":{\"groupName\":\"k8s-app-appingre-64b5836d2b\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"tags\":{\"Environment\":\"test\"},\"ingress\":[{\"ipProtocol\":\"tcp\",\"fromPort\":80,\"toPort\":80,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]}]}}},\"AWS::ElasticLoadBalancingV2::Listener\":{\"80\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":80,\"protocol\":\"HTTP\",\"defaultActions\":[{\"type\":\"fixed-response\",\"fixedResponseConfig\":{\"contentType\":\"text/plain\",\"statusCode\":\"404\"}}],\"tags\":{\"Environment\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::ListenerRule\":{\"80:1\":{\"spec\":{\"listenerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::Listener/80/status/listenerARN\"},\"priority\":1,\"actions\":[{\"type\":\"forward\",\"forwardConfig\":{\"targetGroups\":[{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/app/app-ingress-api-http:80/status/targetGroupARN\"}}]}}],\"conditions\":[{\"field\":\"path-pattern\",\"pathPatternConfig\":{\"values\":[\"/*\"]}}],\"tags\":{\"Environment\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-app-appingre-df49e963f5\",\"type\":\"application\",\"scheme\":\"internet-facing\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-0257383f56bbc4810\"},{\"subnetID\":\"subnet-0b8b2e282788f64bc\"}],\"securityGroups\":[{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"}],\"tags\":{\"Environment\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::TargetGroup\":{\"app/app-ingress-api-http:80\":{\"spec\":{\"name\":\"k8s-app-apihttp-ee37d33f98\",\"targetType\":\"instance\",\"port\":30867,\"protocol\":\"HTTP\",\"protocolVersion\":\"HTTP1\",\"healthCheckConfig\":{\"port\":\"traffic-port\",\"protocol\":\"HTTP\",\"path\":\"/\",\"matcher\":{\"httpCode\":\"200\"},\"intervalSeconds\":15,\"timeoutSeconds\":5,\"healthyThresholdCount\":2,\"unhealthyThresholdCount\":2},\"tags\":{\"Environment\":\"test\"}}}},\"K8S::ElasticLoadBalancingV2::TargetGroupBinding\":{\"app/app-ingress-api-http:80\":{\"spec\":{\"template\":{\"metadata\":{\"name\":\"k8s-app-apihttp-ee37d33f98\",\"namespace\":\"app\",\"creationTimestamp\":null},\"spec\":{\"targetGroupARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::TargetGroup/app/app-ingress-api-http:80/status/targetGroupARN\"},\"targetType\":\"instance\",\"serviceRef\":{\"name\":\"api-http\",\"port\":80},\"networking\":{\"ingress\":[{\"from\":[{\"securityGroup\":{\"groupID\":{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"}}}],\"ports\":[{\"protocol\":\"TCP\"}]}]}}}}}}}}"}
{"level":"info","ts":1623379852.083325,"logger":"controllers.ingress","msg":"creating securityGroup","resourceID":"ManagedLBSecurityGroup"}
{"level":"info","ts":1623379852.2566178,"logger":"controllers.ingress","msg":"created securityGroup","resourceID":"ManagedLBSecurityGroup","securityGroupID":"sg-080bffd697dc0ab27"}
{"level":"info","ts":1623379852.4109924,"msg":"authorizing securityGroup ingress","securityGroupID":"sg-080bffd697dc0ab27","permission":[{"FromPort":80,"IpProtocol":"tcp","IpRanges":[{"CidrIp":"0.0.0.0/0","Description":""}],"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":80,"UserIdGroupPairs":null}]}
{"level":"info","ts":1623379852.5575335,"msg":"authorized securityGroup ingress","securityGroupID":"sg-080bffd697dc0ab27"}
{"level":"info","ts":1623379852.6474898,"logger":"controllers.ingress","msg":"creating targetGroup","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80"}
{"level":"info","ts":1623379852.9693868,"logger":"controllers.ingress","msg":"created targetGroup","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/k8s-app-apihttp-ee37d33f98/8e8cf427880a005f"}
{"level":"info","ts":1623379853.0108032,"logger":"controllers.ingress","msg":"creating loadBalancer","stackID":"app/app-ingress","resourceID":"LoadBalancer"}
{"level":"info","ts":1623379853.6124952,"logger":"controllers.ingress","msg":"created loadBalancer","stackID":"app/app-ingress","resourceID":"LoadBalancer","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:loadbalancer/app/k8s-app-appingre-df49e963f5/e1e08ff54e34da78"}
{"level":"info","ts":1623379853.6379986,"logger":"controllers.ingress","msg":"creating listener","stackID":"app/app-ingress","resourceID":"80"}
{"level":"info","ts":1623379853.7017045,"logger":"controllers.ingress","msg":"created listener","stackID":"app/app-ingress","resourceID":"80","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:listener/app/k8s-app-appingre-df49e963f5/e1e08ff54e34da78/7896e4ee12889f1d"}
{"level":"info","ts":1623379853.7323081,"logger":"controllers.ingress","msg":"creating listener rule","stackID":"app/app-ingress","resourceID":"80:1"}
{"level":"info","ts":1623379853.7993383,"logger":"controllers.ingress","msg":"created listener rule","stackID":"app/app-ingress","resourceID":"80:1","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:listener-rule/app/k8s-app-appingre-df49e963f5/e1e08ff54e34da78/7896e4ee12889f1d/2fc270d382a475d1"}
{"level":"info","ts":1623379853.7995286,"logger":"controllers.ingress","msg":"creating targetGroupBinding","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80"}
{"level":"info","ts":1623379853.836185,"logger":"controllers.ingress","msg":"created targetGroupBinding","stackID":"app/app-ingress","resourceID":"app/app-ingress-api-http:80","targetGroupBinding":{"namespace":"app","name":"k8s-app-apihttp-ee37d33f98"}}
{"level":"info","ts":1623379853.989902,"logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"app/app-ingress"}
{"level":"info","ts":1623379854.0531545,"msg":"authorizing securityGroup ingress","securityGroupID":"sg-03aa126eb1aaeacc5","permission":[{"FromPort":0,"IpProtocol":"tcp","IpRanges":null,"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":65535,"UserIdGroupPairs":[{"Description":"elbv2.k8s.aws/targetGroupBinding=shared","GroupId":"sg-080bffd697dc0ab27","GroupName":null,"PeeringStatus":null,"UserId":null,"VpcId":null,"VpcPeeringConnectionId":null}]}]}
{"level":"info","ts":1623379854.1990266,"msg":"authorized securityGroup ingress","securityGroupID":"sg-03aa126eb1aaeacc5"}
{"level":"info","ts":1623379854.3660266,"msg":"registering targets","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/k8s-app-apihttp-ee37d33f98/8e8cf427880a005f","targets":[{"AvailabilityZone":null,"Id":"i-08b9e38bcf885a07a","Port":30867},{"AvailabilityZone":null,"Id":"i-0cdcd5bd12476c990","Port":30867}]}
{"level":"info","ts":1623379854.5757735,"msg":"registered targets","arn":"arn:aws:elasticloadbalancing:us-east-1:111111111111:targetgroup/k8s-app-apihttp-ee37d33f98/8e8cf427880a005f"}
Here are the logs for the other pod in the AWS Load Balancer Controller deployment:
~/git/github/django-cdk$ kubectl logs -n kube-system goeksstackdjangoekssamplealbingresscontroller48a16415-aws-q9nmg
{"level":"info","ts":1623379840.7888832,"msg":"version","GitVersion":"v2.2.0","GitCommit":"68c417a7ea37ff153f053d9ffef1cc5c70d7e211","BuildDate":"2021-05-14T21:49:05+0000"}
{"level":"info","ts":1623379840.8260994,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":1623379840.8316746,"logger":"setup","msg":"adding health check for controller"}
{"level":"info","ts":1623379840.8317719,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-v1-pod"}
{"level":"info","ts":1623379840.8318048,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/mutate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379840.83182,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-elbv2-k8s-aws-v1beta1-targetgroupbinding"}
{"level":"info","ts":1623379840.831849,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-networking-v1beta1-ingress"}
{"level":"info","ts":1623379840.8319223,"logger":"setup","msg":"starting podInfo repo"}
I0611 02:50:42.832101 1 leaderelection.go:242] attempting to acquire leader lease kube-system/aws-load-balancer-controller-leader...
{"level":"info","ts":1623379842.8324547,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1623379842.9327362,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"}
{"level":"info","ts":1623379842.9332633,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}
{"level":"info","ts":1623379842.9333658,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443}
{"level":"info","ts":1623379842.9334989,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"}
I replaced my account ID with 111111111111.
Here is the ALB DNS name from my CDK Stack's CfnOutput:
k8s-app-appingre-a5bb1f9208-1217069225.us-east-1.elb.amazonaws.com
and here is the DNS name for the ALB from the EC2 > Load Balancers console:
k8s-app-appingre-df49e963f5-842078657.us-east-1.elb.amazonaws.com
Use this "Ingress.yaml" as example:
# Annotations Reference: https://kubernetes-sigs.github.io/aws-alb-ingress-controller/guide/ingress/annotation/
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: eks-microservices-demo
labels:
app: usermgmt-restapp
annotations:
# Ingress Core Settings
kubernetes.io/ingress.class: "alb"
alb.ingress.kubernetes.io/scheme: internet-facing
# Health Check Settings
alb.ingress.kubernetes.io/healthcheck-protocol: HTTP
alb.ingress.kubernetes.io/healthcheck-port: traffic-port
alb.ingress.kubernetes.io/healthcheck-interval-seconds: '15'
alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '5'
alb.ingress.kubernetes.io/success-codes: '200'
alb.ingress.kubernetes.io/healthy-threshold-count: '2'
alb.ingress.kubernetes.io/unhealthy-threshold-count: '2'
## SSL Settings
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:ddddddddddddd
#alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-1-2017-01 #Optional (Picks default if not used)
# SSL Redirect Setting
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
# External DNS - For creating a Record Set in Route53
external-dns.alpha.kubernetes.io/hostname: users.skycomposer.net
spec:
rules:
- http:
paths:
- path: /* # SSL Redirect Setting
backend:
serviceName: ssl-redirect
servicePort: use-annotation
- path: /*
backend:
serviceName: usermgmt-restapp-nodeport-service
servicePort: 8095
# Important Note-1: In path based routing order is very important, if we are going to use "/*", try to use it at the end of all rules.
P.S. Warning! It will only work with managed AWS EKS cluster, or any other cluster, which supports ALB Ingress Controller.
P.P.S. With K3S Kubernetes cluster, I use Traefik Ingress Controller and have to manually create Route 53 CNAME Record with the name of domain and the value of Load Balancer DNS Name. Unfortunately, I don't know how to automate this process.
The full source code example of AWS K3S Kuberneter Cluster with Traefik Ingress Controller, with step-by-step instructions, can be found here: https://github.com/skyglass/customer-management-keycloak