I have tried to make a redis cluster in k8s environment using "NodePort" type of service. More specifically, I want to compose a redis cluster across two different k8s cluster.
When I used LoadBalancer(External IP) for service type, cluster was made successfully. The problem is NodePort.
After I command redis-cli --cluster create, it stucks on "Waiting for the cluster to join"
Below is the logs of cluster create command. I deployed 4 leader pods and 4 slave pods with individual nodeport service.
root@redis-leader00-5fc546c4bd-28x8w:/data# redis-cli -a mypassword --cluster create --cluster-replicas 1 \
> 192.168.9.194:30030 192.168.9.199:30031 192.168.9.194:30032 192.168.9.199:30033 \
> 192.168.9.199:30030 192.168.9.194:30031 192.168.9.199:30032 192.168.9.194:30033
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
>>> Performing hash slots allocation on 8 nodes...
Master[0] -> Slots 0 - 4095
Master[1] -> Slots 4096 - 8191
Master[2] -> Slots 8192 - 12287
Master[3] -> Slots 12288 - 16383
Adding replica 192.168.9.199:30030 to 192.168.9.194:30030
Adding replica 192.168.9.194:30033 to 192.168.9.199:30031
Adding replica 192.168.9.199:30032 to 192.168.9.194:30032
Adding replica 192.168.9.194:30031 to 192.168.9.199:30033
M: 94bf3c6760e6b3a91c408eda497822b4961e8d82 192.168.9.194:30030
slots:[0-4095] (4096 slots) master
M: 31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031
slots:[4096-8191] (4096 slots) master
M: 0738d1e1a677352fc3b0b3600a67d837b795fa8a 192.168.9.194:30032
slots:[8192-12287] (4096 slots) master
M: 7dd7edbfab6952273460778d1f140b0716118042 192.168.9.199:30033
slots:[12288-16383] (4096 slots) master
S: 17e044681319d0a05bd92deeb4ead31c0cd468e2 192.168.9.199:30030
replicates 94bf3c6760e6b3a91c408eda497822b4961e8d82
S: 9c9e47ec566ac781e8e3dcb51398a27d1da71004 192.168.9.194:30031
replicates 7dd7edbfab6952273460778d1f140b0716118042
S: b8f7028b56f96565a91fdb442c94fbedcee088c2 192.168.9.199:30032
replicates 0738d1e1a677352fc3b0b3600a67d837b795fa8a
S: e4c9ffdf67e2b2ef9f840618110738358bde52d5 192.168.9.194:30033
replicates 31f4a9604b15109316f91956aa4a32b0c6952a4d
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
...........................................
The weird point is that the other redis containers received the signal but there is no progress. Below is the logs of other redis containers.
.... // Some logs for initializing redis
1:M 20 Jan 2022 06:09:12.055 * Ready to accept connections
1:M 20 Jan 2022 06:13:41.263 # configEpoch set to 5 via CLUSTER SET-CONFIG-EPOCH
I thought the communication was successful but gossip port didn't work properly. So, I modified redis.conf and set cluster-announce-bus-port but it also didn't work.
How can I compose a redis cluster using nodeport type of service?
Please refer the one of .yaml files
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-cluster-leader00
namespace: redis
labels:
app: redis-cluster
leader: "00"
data:
fix-ip.sh: |
#!/bin/sh
CLUSTER_CONFIG="/data/nodes.conf"
if [ -f ${CLUSTER_CONFIG} ]; then
if [ -z "${HOST_IP}" ]; then
echo "Unable to determine Pod IP address!"
exit 1
fi
echo "Updating my IP to ${HOST_IP} in ${CLUSTER_CONFIG}"
sed -i.bak -e "/myself/ s/[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}/${HOST_IP}/" ${CLUSTER_CONFIG}
fi
exec "$@"
redis.conf: |+
bind 0.0.0.0
cluster-enabled yes
cluster-require-full-coverage no
cluster-node-timeout 15000
cluster-config-file /data/nodes.conf
cluster-migration-barrier 1
appendonly no
save ""
protected-mode no
requirepass "mypassword"
masterauth "mypassword"
cluster-announce-ip 192.168.9.194
cluster-announce-port 30030
cluster-announce-bus-port 31030
---
apiVersion: v1
kind: Service
metadata:
name: redis-leader00
namespace: redis
labels:
app: redis
role: leader
tier: backend
leader: "00"
spec:
ports:
- port: 6379
targetPort: 6379
nodePort: 30030
name: client
- port: 16379
targetPort: 16379
nodePort: 31030
name: gossip
selector:
app: redis
role: leader
tier: backend
leader: "00"
type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-leader00
namespace: redis
labels:
app: redis
role: leader
tier: backend
leader: "00"
spec:
replicas: 1
selector:
matchLabels:
app: redis
leader: "00"
template:
metadata:
labels:
app: redis
role: leader
tier: backend
leader: "00"
spec:
containers:
- name: leader
image: docker.io/redis:6.0.5
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
name: client
- containerPort: 16379
name: gossip
volumeMounts:
- name: conf
mountPath: /conf
readOnly: false
args: ["--requirepass", "mypassword"]
command: ["/conf/fix-ip.sh", "redis-server", "/conf/redis.conf"]
env:
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
volumes:
- name: conf
configMap:
name: redis-cluster-leader00
defaultMode: 0755
Also see nodes.conf file in container after I commanded redis-cli create
root@redis-leader01-87ccb466-bsnq4:/data# cat nodes.conf
31f4a9604b15109316f91956aa4a32b0c6952a4d 192.168.9.199:30031@31031 myself,master - 0 0 2 connected 4096-8191
vars currentEpoch 2 lastVoteEpoch 0
There was someone else who had already found the problem. This link is the issue he made. https://github.com/redis/redis/issues/6432
The cause of problem is that CLUSTER MEET is not use cluster-announce-bus-port. Actually, there is even no variable for storing cluster-announce-bus-port in clusterManagerNode struct.
There is already PR for this problem. https://github.com/redis/redis/pull/6442
It's not merged or closed yet so this is the ongoing problem...