Keys in Redis cluster cannot be deleted (and have empty value)

We have a Redis cluster (3 master, 3 slave) and are seeing a large number of keys on one of the master nodes (and related slaves) which appear to be empty and cannot be deleted.

If I connect to the master node which has a large number of entries (as determined by DBSIZE), I can SCAN for and see the keys:

--> redis-cli -h 192.168.100.81 -p 6381
192.168.100.81:6381> scan 0 match mykey-* count 10
1) "2359296"
2)  1) "mykey-1be333a7"
    2) "mykey-e85a9d31"
    3) "mykey-d9162eff"
    4) "mykey-41d12fd8"
    5) "mykey-a6e755d3"
    6) "mykey-c2aa1eaa"
    7) "mykey-c0597cac"
    8) "mykey-10e69376"
    9) "mykey-7263aef0"
   10) "mykey-7fa9de50"

However, if I try and GET the value of a key, it shows it has moved:

192.168.100.81:6381> get mykey-1be333a7
(error) MOVED 8301 192.168.3.107:6380

If I connect to the node to which the key has moved, I am not able to GET or DEL the value:

--> redis-cli -h 192.168.3.107 -p 6380
192.168.3.107:6380> get mykey-1be333a7
(nil)
192.168.3.107:6380> del mykey-1be333a7
(integer) 0

I am also unable to GET or DEL the value using the cluster (-c) flag for redis-cli:

--> redis-cli -h 192.168.100.81 -p 6381 -c get mykey-1be333a7
(nil)
--> redis-cli -h 192.168.100.81 -p 6381 -c del mykey-1be333a7
(integer) 0
--> redis-cli -h 192.168.3.107 -p 6380 -c get mykey-1be333a7
(nil)
--> redis-cli -h 192.168.3.107 -p 6380 -c del mykey-1be333a7
(integer) 0

What can I do to remove these types of keys?

Solution

Interesting question!

How to reproduce the problem

The following is a scenario that will reproduce your problem:

You create a standalone Redis instance, and set some data into it. However, one day, you configure this standalone Redis as a member of a Redis Cluster, without flushing the old data or moving the old data to the right node of the cluster. Let's call these data as dirty data.

In this scenario, you can SCAN all keys on this instance, including the dirty data. However, you cannot read or write these dirty data, since your Redis is in cluster mode, it will redirect your request to the right node, which doesn't have such data.

How to solve the problem

In order to remove these dirty data, you should reconfigure your Redis instance into standalone mode, i.e. cluster-enabled no, and delete dirty data in standalone mode.

Then you can make it join the cluster again.