Passing an ARN from ConfigMap tp TargetGroupBinding in Kubernetes

Question

I need to do a Target Group Binding in AWS. I am creating almost all resources using Terraform (EKS cluster, nodes, other AWS services). On top of it I am having a Kubernetes cluster of course and code written in K8s yamls.

I am creating a service like f.e. nginx proxy which is meant to be dong its proxying job.

What I want to achieve is to bind the proxy service

kind: Service
metadata:
  name: nginx-proxy-service
  namespace: nginx-proxy
spec:
  selector:
    app: nginx-proxy
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443

with a target group created in Terraform

resource "aws_lb_target_group" "nginx-proxy" {
  name        = "${var.environment}-proxy-tg"
  port        = 443
  protocol    = "HTTPS"
  vpc_id      = var.vpc_id
  target_type = "ip"
}

There is a CRD TargetGroupBinding that I can use for this purpose, but I need to pass the TargetGroup ARN to it

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: nginx-proxy-tgb
  namespace: nginx-proxy
spec:
  serviceRef:
    name: nginx-proxy-service
    port: 443
  targetGroupARN: $(TARGETGROUP_ARN)

Firstly I thought that I can use a ConfigMap, but that TargetGroupBinding doesn't understand ConfigMaps. Then I thought that I can use Kustomize, but I cannot figure out how (if it is possible) to pass a value from a ConfigMap that contains that value,

$ kubectl describe configmap proxy-cm
Name:         proxy-cm
Namespace:    nginx-proxy
Labels:       <none>
Annotations:  <none>

Data
====
targetgroup_arn:
----
arn:aws:elasticloadbalancing:eu-west-1:<somevaluehere>:targetgroup/beta-proxy-tg/<somevaluethere>

BinaryData
====

Events:  <none>

to a Kustomize, and then Kustomize can use it and replace proper fields in my TargetGroupBinding.

Do you have any ideas how can I get marry those two things? Seems pretty common pattern, but as a newbie in Terraform and K8s I cannot figure it out.

Answer 1

I want to use Terraform to create infrastructure obcjets (EKS clusters, nodes, Load Balancers, Target groups and so on). I want to use kubernetes/Kustomize/Helm code to create K8s objects like deployments, services, statefulsets ad so on. I've almost done that using Helm chart approach

Terraform object creation:

resource "aws_lb_target_group" "nginx-proxy" {
  name        = "${var.environment}-proxy-tg"
  port        = 443
  protocol    = "HTTPS"
  vpc_id      = var.vpc_id
  target_type = "ip"
}

resource "kubernetes_config_map" "proxy_configmap" {
  metadata {
    name      = "proxy-cm"
    namespace = "nginx-proxy"
  }
  data = {
    targetgroup_arn  = "${aws_lb_target_group.nginx-proxy.arn}"
    lb-name = "${aws_lb.eks_alb.name}"
  }
  depends_on = [kubernetes_namespace.nginx-proxy-namespace]
}

_helpers.tpl

{{- define "awsLBTargetGroupArn" -}}
{{- (lookup "v1" "ConfigMap" "nginx-proxy" "proxy-cm").data.targetgroup_arn }}
{{- end -}}

Helm templates:

apiVersion: v1
kind: Service
metadata:
  name: nginx-proxy-service
  namespace: nginx-proxy
spec:
  selector:
    app: nginx-proxy
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: nginx-proxy-tgb
  namespace: nginx-proxy
spec:
  serviceRef:
    name: nginx-proxy-service
    port: 443
  targetGroupARN: {{ include "awsLBTargetGroupArn" . }}

I can see that kubectl shows me that it is being binded

✗ kubectl describe TargetGroupBinding nginx-proxy-tgb
Name:         nginx-proxy-tgb
Namespace:    nginx-proxy
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: nginx-proxy
              meta.helm.sh/release-namespace: nginx-proxy
API Version:  elbv2.k8s.aws/v1beta1
Kind:         TargetGroupBinding
(...)



Ip Address Type:  ipv4
  Service Ref:
    Name:            nginx-proxy-service
    Port:            443
  Target Group ARN:  arn:aws:elasticloadbalancing:eu-west-1:<somenumbershere>:targetgroup/beta-proxy-tg/bba0b9519459370e
  Target Type:       ip
Status:
  Observed Generation:  1
Events:
  Type    Reason                  Age                From                Message
  ----    ------                  ----               ----                -------
  Normal  SuccessfullyReconciled  48m (x3 over 83m)  targetGroupBinding  Successfully reconciled

But I can't see the nodes being attached in fact (solved and described down the post.)

✗ aws elbv2 describe-target-health  --target-group-arn arn:aws:elasticloadbalancing:eu-west-1:<somenumbershere>:targetgroup/beta-proxy-tg/bba0b9519459370e
----------------------
|DescribeTargetHealth|
+--------------------+