I am experimenting with a kubernetes cluster on aws.
At the end of the day, I want to expose 2 urls:
production.somesite.comstaging.somesite.comWhen exposing 1 url, things (at least in the cloud landscape) seem to be easy.
You make the service LoadBalancer type --> aws provisions an ELB --> you assign an A type alias record (e.g. whatever.somesite.com) to ELB's dns name and boom, there is your service publicly available via the hostname you like.
I assume one easy (and I guess not best-pracise-wise) way of going about this is to expose 2 ELBs.
Is Ingress the (good) alternative?
If so, what is the Route53 record I should create?
For what that matters (and in case this may be a dealbreaker for Ingress):
production.somesite.com will be publicly availablestaging.somesite.com will have restrictive accesThis is a great question! I have to say that @whites11 answer was perfect in 2018, and is still pretty much on point. But five years later, we have a few more choices available to us, so let's talk about that!
First, the simplest thing to do would probably be to expose each URL (production.somesite.com and staging.somesite.com) with its own LoadBalancer Service. Then, for each Service, retrieve its EXTERNAL-IP (which, on AWS, will actually be a name, not an IP address), and set up a CNAME record from e.g. production.somesite.com pointing to the corresponding ELB.
Pros: simplicity, as you don't need to install anything else. You're also decoupling all DNS operations (your domain names could be hosted with Route53 or with just anything else).
Cons: price, since you're paying for one ELB per Service exposed this way. For a couple of URLs, it doesn't matter much; but if you have dozens or even hundreds of them, that becomes a concern. Furthermore, each service requires an additional manual step to set up the corresponding DNS record.
The next option is to use an Ingress controller. An Ingress controller is essentially a load balancer that "speaks" HTTP (technically, it parses HTTP requests) and as such, is capable of doing content-based routing. That's a fancy way to say that the Ingress controller can look at the content of the request, and depending on the host header (production.somesite.com or staging.somesite.com) or even the URI (/hello.html or /static/theme.css or /api/v1/users/42) it can send that request to a different Kubernetes Service.
If we decide to go that route (no pun intended), it gets tricky because we have so many more options!
Roughly speaking, we have Ingress controllers like Ingress NGINX or Traefik that can be deployed on almost any kind of Kubernetes cluster - cloud, on premises, home labs, even local clusters like KinD or Minikube.
And then we have cloud-specific Ingress controllers, like AWS Load Balancer Controller (formerly known as "ALB Ingress") or GKE Ingress.
The advantage of the cloud-specific Ingress controllers is that they can offer a shorter path for our traffic, as it will do something like this:
HTTP client → Ingress LB → HTTP server Pod
While using e.g. NGINX or Traefik in the cloud will typically do something like that:
HTTP client → Cloud LB → Ingress Pod → HTTP server Pod
At a first glance, it looks like saving one hop would be a pretty big deal; but it's not always the case. This is internal traffic, which means that it's usually pretty fast (say, 1ms) and pretty cheap (a fraction of the cost of external traffic). If shaving 1 millisecond of latency off your HTTP response time matters to you, or if you have a huge amount of internal HTTP traffic (i.e. not coming from external HTTP clients), then by all means, check out your cloud-specific Ingress controller. On the other hand, setting the AWS Load Balancer Controller can be significantly more complex, involving extra IAM and network configuration, and it requires the use of the AWS VPC CNI (in other words: not Calico or the increasingly popular Cilium). Meanwhile, the "generic" Ingress controllers can be installed with a one-liner command that will "just work" and be compatible with pretty much any Kubernetes cluster and CNI plugin you might be using.
Note that we said "in the cloud" earlier. If you're on premises, it's quite possible (and common) to run the Ingress controller using e.g. a DaemonSet and hostPort, and get back to a traffic path like the following:
HTTP client → Ingress Pod → HTTP server Pod
For a simple scenario with only two URLs, we could probably wrap up here. But a lot of folks deploying Kubernetes are leveraging it to run way more applications than that, and possibly add new ones regularly. This is where another project might be worth mentioning: ExternalDNS.
ExternalDNS automatically manages DNS records, so that when you create e.g. an Ingress resource for production.somesite.com, it creates the corresponding DNS record pointing to the IP address of its Ingress controller. This removes a manual step in our original process, which is particularly convenient once you have not just a couple of URLs, but dozens of them, and/or if you're constantly adding, removing, or even moving these URLs from one cluster to another: ExternalDNS will make sure that they always point to the right place. ExternalDNS supports Route53 as well as more than 30 other DNS providers (both SAAS and self-hosted servers).
Another thing that you can do is leverage a wildcard DNS record. For instance, you could map *.wild.somesite.com to your Ingress controller LoadBalancer address. Then, whenever you create an Ingress for <anything>.wild.somesite.com, you won't need to add a new DNS record, because it'll already be covered by the wildcard record. This means that you won't need ExternalDNS, and it may reduce the time it takes for a website to be reachable (since some DNS providers might take a minute or a few before records are online).
Additionally, if you want to expose your apps and website over HTTPS, you can leverage cert-manager in combination with a service like Let's Encrypt or ZeroSSL to automatically generate key pairs and get a valid TLS certificate for each of your apps.
One last thing! Since the original question asked staging to have a restricted access - while this is not something that can be done in a standard way with a Kubernetes Ingress resource, most Ingress controllers support vendor-specific ways to add custom configurations. You can check the NGINX Ingress documentation for an example of that feature, allowing (almost) any arbitrary NGINX configuration directive. Yay!