I have setup a Redis-Cluster with version 3.0.5 of Redis-Server (Ubuntu 14.04)
For simplicity, we shall ignore replication. I have three redis instances running on localhost, ports 7001, 7002 and 7003. They are all made masters of a cluster with this command
redis-trib.rb create 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003
I like the idea of using twemproxy
twemproxy-config.yml
twem-1:
listen: 127.0.0.1:22121
hash: fnv1a_64
distribution: ketama
redis: true
preconnect: true
servers:
- 127.0.0.1:7001:1
- 127.0.0.1:7002:1
- 127.0.0.1:7003:1
I initialize twemproxy with nutcracker -c twemproxy-config.yml -d and then I can access twemproxy with redis-cli -h 127.0.0.1 -p 22121
Please have a look at this input and output
127.0.0.1:22121> set hello 4542342342424
OK
127.0.0.1:22121> set goodbye 345353535545
(error) MOVED 9354 159.203.136.204:7002
127.0.0.1:22121> get hello
"4542342342424"
127.0.0.1:22121> get goodbye
(error) MOVED 9354 159.203.136.204:7002
I am concerned that this may not be working properly. If I bypass twemproxy and connect using redis-cli -c -h 127.0.0.1 -p 7001, I can see automatic forwarding taking place. Like so;
127.0.0.1:7001> get hello
"4542342342424"
127.0.0.1:7001> get goodbye
-> Redirected to slot [9354] located at 127.0.0.1:7002
(nil)
127.0.0.1:7002> set goodbye 3240923842094840
OK
127.0.0.1:7002> get goodbye
"3240923842094840"
Reference
Interesting read at code.hootsuite.com about use of twemproxy (about halfway down page)
Ultimate aim
My ultimate aim is to use the redis cluster for storing PHP session data across multiple webservers behind a load balancer. In php.ini I would have session.save_handler = redis and session.save_path = tcp://127.0.0.1:22121 (the twemproxy instance would run on each web server). The PHP session configuration bit isn't set up yet.
I hope I am making sense. Am I using the correct hashing code? I really would like to see twemproxy giving back OK instead of MOVED.
Thanks!
Thanks to the answer from @the-real-bill I have setup redis on two nodes with standard redis running port 6380 and sentinel running port 16380, one is master the other is slave. Watching the logs, everything looks good.
I've looked at predis and phpredis
This is the bit I'm still a little unsure about and I am sure given more time I will work it out. Queries can only be written to a master, and queries can only be read from a slave. We've got Sentinel promoting one or the other depending on availabbility - how does the session handler know which one it can write to? Surely I need to provide two IP adddresses..
Redis Cluster is a client-aware pattern. In Redis Cluster the client always most connect to the "correct" node for a given key. The MOVED reply is telling the client which node services that key. Initially a client is expected to pull the current topology on connect and then update if it gets a MOVED. This is the highest performing pattern as there is no proxy involved.
However, this means you can not use any existing proxy in front of a Redis Cluster setup. As you have seen Twemproxy simply proxies certain commands, and refuses others. In order for Twemproxy to handle this it would need to implement the cluster API and do everything the client does. This is does not do, nor should you expect it to do such soon. Any other proxy such as nginx or HAProxy would need to do the same thing (hypothetically with openresty or a custom module you could do this in Nginx easier than the others).
Further, last I checked the PHP clients do not support Redis Cluster.
However, for your use case I doubt you actually need a Redis Cluster. Configuring Redis in a Pod (master+slave(s)) with Sentinel to manage failover combined with client-side Sentinel support would likely meet your needs quite well. I believe PRedis has support but I am sure phpredis does not.
If you have your heart set on the complexity of running a shared cluster, then use regular Redis + Sentinel behind Twemproxy and let Twemproxy connect via the sharing algorithm you tell it to use. This is the other rub with trying to use Tweproxy - it does what Redis Cluster does but as a proxy. So in effect you'd be trying to shard the data set twice. I' sure you can imagine ways in which that is catastrophic. ;)
If you can't get proper support or don't want Twemproxy managing sharing for you, your final option (short of adding client support yourself) is to set up a proxy which is configured by sentinel. For example you could use HAProxy configured to point to the master. On a failover Sentinel can execute a script for you.
This script could then update your HAProxy configuration and restart it for you, thus ensuring proper reconnect behavior. There are many ways to do this and a Google search on it, combined with the Redis Sentinel docs and applying them to your environment would show the path for you.