Search code examples
redisconsulfailover

Failover redis with Consul DNS

How does Consul resolve the new redis master ip address after the old redis master get takedown ? For example : I did while true; do dig redis.service.google.consul +short; sleep 2; done the response is 

192.168.248.43
192.168.248.41
192.168.248.42

192.168.248.41
192.168.248.42
192.168.248.43

...

My expectation is it's only resolve to 192.168.248.41 because it's master. But when the master is down, consul should resolve to 192.168.248.42 or 192.168.248.43, according which one is master

Here is my consul services config in 192.168.248.41

...
"services": [
        {
            "id": "redis01-xxx",
            "name": "redis",
            "tags": ["staging"],
            "address": "192.168.248.41",
            "port": 6379
        }
    ]
...

Here is my consul services config in 192.168.248.42

...
"services": [
        {
            "id": "redis01-xxx",
            "name": "redis",
            "tags": ["staging"],
            "address": "192.168.248.42",
            "port": 6379
        }
    ]
...

And same thing with 192.168.248.43.

My expectation is like this video https://www.youtube.com/watch?v=VHpZ0o8P-Ts

when i do dig, the consul will resolve to only one IP address (master). When the master is down and redis sentinel selects the new master. The consul will resolve to new redis master IP address.

I am very new in Consul. So, i am really appreciate if someone can give short example and suggestion feature of consul, so i can catch it faster.

Solution

  • When you register multiple services with the same "name", they will appear under <name>.service.consul (if they are healthy of course). In your case, you didn't define any health-checks for your redis services, that's why you see all of the service IPs in your DNS query response.

    For use-case, where you only need IP of the Redis master, I would recommend registering service with a health check . I'm not familiar how one would query Redis, to find out if the node is the master. But you can use for an example script check, where your script would return 0, if current Redis node was elected master by the Redis sentinel. Consul will automatically recognize that service instance as healthy, and you will see only it's IP in the dig redis.service.consul query to Consul DNS interface.