Search code examples
kubernetesredisredis-sentinel

Redis Sentinel doesn't auto discover new slave instances

I've deployed the redis helm chart on k8s with Sentinel enabled.

I've set up the Master-Replicas with Sentinel topology, it means one master and two slaves. Each pod is running both the redis and sentinel container successfully:

NAME             READY   STATUS    RESTARTS   AGE     IP             NODE
my-redis-pod-0   2/2     Running   0          5d22h   10.244.0.173   node-pool-u
my-redis-pod-1   2/2     Running   0          5d22h   10.244.1.96    node-pool-j
my-redis-pod-2   2/2     Running   0          3d23h   10.244.1.145   node-pool-e

Now, I've a python script that connects to redis and discovers the master by passing it the pod's ip.

sentinel = Sentinel([('10.244.0.173', 26379),
                     ('10.244.1.96',26379),
                     ('10.244.1.145',26379)],
                  sentinel_kwargs={'password': 'redispswd'})

host, port = sentinel.discover_master('mymaster')
redis_client = StrictRedis(
            host=host,
            port=port,
            password='redispswd')

Let's suposse the master node is on my-redis-pod-0, when I do kubectl delete pod to simulate a problem that leads me to loss the pod, Sentinel will promote one of the others slaves to master and kubernetes will give me a new pod with redis and sentinel.

NAME             READY   STATUS    RESTARTS   AGE     IP             NODE
my-redis-pod-0   2/2     Running   0          3m      10.244.0.27    node-pool-u
my-redis-pod-1   2/2     Running   0          5d22h   10.244.1.96    node-pool-j
my-redis-pod-2   2/2     Running   0          3d23h   10.244.1.145   node-pool-e

The question is, how can I do to tell Sentinel to add this new ip to the list automatically (without code changes)?

Thanks!

Solution

  • Instead of using IPs, you may use the dns entries for a headless service.

    A headless service is created by explicitly specifying

    ClusterIP: None

    Then you will be able to use the dns entries as under, where redis-0 will be the master

    #syntax
pod_name.service_name.namespace.svc.cluster.local

#Example
redis-0.redis.redis.svc.cluster.local
redis-1.redis.redis.svc.cluster.local
redis-2.redis.redis.svc.cluster.local

    References: What is a headless service, what does it do/accomplish, and what are some legitimate use cases for it?

    https://www.containiq.com/post/deploy-redis-cluster-on-kubernetes