Search code examples
pythonredispipelineredis-cluster

RedisCluster MGET with pipeline


I am trying to perform an MGET operation on my Redis with the pipeline to increase performance. I have tried doing MGET in one go as well as as a batch flow

from rediscluster import RedisCluster
ru = RedisCluster(startup_nodes=[{"host": "somecache.aws.com", "port": "7845"}], 
        decode_responses=True, 
        skip_full_coverage_check=True)
    
pipe = ru.pipeline()        
# pipe.mget(keys)
    
for i in range(0, len(keys), batch_size):
    temp_list = keys[i:i + batch_size]
    pipe.mget(temp_list)

resp = pipe.execute()

So far I am getting the error

raise RedisClusterException("ERROR: Calling pipelined function {0} is blocked 
when running redis in cluster mode...".format(func.__name__))
rediscluster.exceptions.RedisClusterException: ERROR: 
Calling pipelined function mget is blocked when running redis in cluster mode...

What I want to know is that

  1. Does RedisCluster pipelined MGET?
  2. If not then is there any other lib that I can use to archive this?

Solution

  • Turns out we can not use MGET with the pipeline, below is m final solution

    from rediscluster import RedisCluster
        
    def redis_multi_get(rc: RedisCluster, keys: list):
        pipe = rc.pipeline()
        [pipe.get(k) for k in keys]
        return pipe.execute()
        
    if __name__ == '__main__':
        rc = RedisCluster(startup_nodes=[{"host": host, "port": port}], decode_responses=True, skip_full_coverage_check=True)
        keys = rc.keys(PREFIX + '*')
        cache_hit = redis_multi_get(rc, keys)