I have an application running on Kubernetes, that uses nginx as the ingress controller that created a load balancer in AWS. I noticed that by default the application is open to the World, with 0.0.0.0/32 is added to the inbound rules of the AWS Security group that is attached to the load balancer.
I want to allow only certain IPs to access the application. That makes me use nginx.ingress.kubernetes.io/whitelist-source-range annotation in the ingress controller.
But I wouldn't know the IPs of the entities that must be allowed to access the application beforehand. An upstream process (Jenkins job) that creates certain containers, which try to talk to the application that's running on Kube.
How can I dynamically modify the ingress controller annotation to add and remove IPs without causing any downtime? And No, I do not have a common IP range that I can add. I have several different VPCs which have their own CIDR blocks.
Short answer: you don't put the whitelist annotation on the controller, you put it on the ingress resource. And updating that does not require any downtime.
--
Long answer: Yes, by default, the controller loadbalancer is open to the world, and that is expected. All traffic comes into the ingress controller's loadbalancer and the controller then determines how to route it within the cluster.
It determines this routing via the use of ingress resources. Here is an example:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/whitelist-source-range: 12.0.0.0/8,10.0.0.0/8
name: ingress-whitelist-example
spec:
rules:
- host: somehost.com
http:
paths:
- backend:
service:
name: service-for-somehost
port:
number: 80
tls:
- hosts:
- somehost.com
secretName: tls-secret
The ingress controller will get traffic for (in this example) the domain host somehost.com and it will route the request to the service service-for-somehost in the cluster, on port 80
If a request comes into the controller outside of the ranges 12.0.0.0/8 or 10.0.0.0/8 (as described by the annotation), then the controller will reject the request with a Forbidden error.
The ingress resource is not a resource that needs to be respun or taken down for updates, like Deployments.
The AWS load balancer is external to the cluster. You can, of course, choose to block/whitelist traffic within the AWS cloud before it reaches the cluster and that is fine, but that is not managed within the nginx controller.
Some further reading in Ingresses is available here