How can I set up an Ingress to connect to a ClusterIP service?

Question

Objective

I am have deployed Apache Airflow on AWS' Elastic Kubernetes Service using Airflow's Stable Helm chart. My goal is to create an Ingress to allow others to access the airflow webserver UI via their browser. It's worth mentioning that I am deploying on EKS using AWS Fargate. My experience with Kubernetes is somewhat limited, and I have not set an Ingress myself before.

What I have tried to do

I am currently able to connect to the airflow web-server pod via port-forwarding (like kubectl port-forward airflow-web-pod 8080:8080). I have tried setting the Ingress through the Helm chart (documented here). After which:

Running kubectl get ingress -n dp-airflow I got:

NAME             CLASS    HOSTS                      ADDRESS   PORTS   AGE
airflow-flower   <none>   foo.bar.com             80      3m46s
airflow-web      <none>   foo.bar.com             80      3m46s

Then running kubectl describe ingress airflow-web -n dp-airflow I get:

Name:             airflow-web
Namespace:        dp-airflow
Address:
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  foo.bar.com
                            /airflow   airflow-web:web (<redacted_ip>:8080)
Annotations:                meta.helm.sh/release-name: airflow
                            meta.helm.sh/release-namespace: dp-airflow

I am not sure what did I need to put into the browser, so I have tried using http://foo.bar.com/airflow as well as the cluster endpoint/ip without success.

This is how the airflow webservice service looks like: Running kubectl get services -n dp-airflow, I get:

NAME          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
airflow-web   ClusterIP   <redacted_ip>   <none>        8080/TCP   28m

Other things I have tried

I have tried creating an Ingress without the Helm chart (I am using Terraform), like:

resource "kubernetes_ingress" "airflow_ingress" {
  metadata {
    name = "ingress"
  }

  spec {
    backend {
      service_name = "airflow-web"
      service_port = 8080
    }

    rule {
      http {
        path {
          backend {
            service_name = "airflow-web"
            service_port = 8080
          }
          path = "/airflow"
        }
      }
    }
  }
}

However I was still not able to connect to the web UI. What are the steps that I need to take to set up an Ingress? Which address do I need to use in my browser to connect to the web UI?

I am happy to provide further details if needed.

Answer 1

It sound like you have created Ingress resources. That is a good step. But for those Ingress resources to have any effect, you also need an Ingress Controller than can realize your Ingress to an actual load balancer.

In an AWS environment, you should look at AWS Load Balancer Controller that creates an AWS Application Load Balancer that is configured according your Ingress resources.

Ingress to connect to a ClusterIP service?

First, the default load balancer is classic load balancer, but you probably want to use the newer Application Load Balancer to be used for your Ingress resources, so on your Ingress resources add this annotation:

annotations:
    kubernetes.io/ingress.class: alb

By default, your services should be of type NodePort, but as you request, it is possible to use ClusterIP services as well, when you on your Ingress resource also add this annotation (for traffic mode):

alb.ingress.kubernetes.io/target-type: ip

See the ALB Ingress documentation for more on this.