Search code examples
kubernetesclient-goleader-election

Kubernetes services for all pods and another for only the leader

In Kubernetes, is it possible to have 2 services for a single deployment, one which is "standard" and proxies in front of all ready pods, and a second service which sends traffic only the elected leader? If so how? I am using client-go for leader election. These are layer 4 services.

I know that a service can use labels for a selector, but client-go uses an annotation to mark the leader. Using a service without selectors and creating/removing an endpoint in the leader callbacks seems hacky/buggy. Thank you.

Solution

  • In Kubernetes, is it possible to have 2 services for a single deployment, one which is "standard" and proxies in front of all ready pods, and a second service that sends traffic only the elected leader?

    Yes, but it seems a bit hacky. The way services work is like this:

    Service -> Service labels select endpoints -> Endpoint -> Endpoint labels select pods -> Pods (PodIP)

    1. So you could have your regular "Service" that points to all the pods on your Deployment or StatefulSet which automatically provisions all the Endpoints.

    2. You could also have another set of "Headless Service" + "Endpoint" each manually created individually that you make their labels match with each other and then have that Endpoint manually match with the label of the pod of your choice.

    Now with respect to client-go/leaderelection. It seems like it works using an Endpoint or ConfigMap lock for the leader (The example shows a ConfigMap lock). But, looks like you want to use the Endpoint lock. So this package doesn't work with services or labels, it looks like it just works on Endpoint resources. So essentially if you have 3 nodes and want to find the leader you would have to use 3 manually created Endpoint resources. The one that is the leader will always have the annotation.

    Now how do you tie it to 2) above? As your client elects or selects the leader then you also have to change the endpoint labels so that they match your manually created headless service. (It could be done in your code too)

    You could also just elect to just use Endpoints instead 2) above (No headless service) and have client-go/leaderelection talk directly to the endpoints.

    Another option is to take advantage of StatefulSets and its required headless service. So that service will resolve to the IP addresses of all the replicas in your quorum-based cluster. The leader election would be up to the client package (client-go doesn't seem to support this) which is pretty the case for most quorum based applications (K8s, Zookeeper, Kafka, Etcd, etc, etc); the client is the one that finds who the leader is.

    ✌️