Search code examples
pythonredisbatch-processing

Is there a way for reading a redis list in bulks?

Assume we have a redis set with hundreds thousands elements in it. As smember command does eager-loading, it fetches all of the elements just by this one command and consequently it consumes too much time. I want to know is there a way to read redis data as bulks or maybe as a stream?

Solution

  • Bulks

    Data from Redis Set data structure can be read in bulks using SSCAN command.

    > sadd set a
(integer) 1
> sadd set b
(integer) 1
> sadd set c
(integer) 1

> SMEMBERS set
1) "c"
2) "b"
3) "a"

> SSCAN set 0 COUNT 2
1) "1"                  // <-- cursor for following SSCAN
2) 1) "b"
   2) "a"
> SSCAN set 1 COUNT 2   // <-- cursor from previous SSCAN
1) "0"                  // <-- cursor=0 indicating complete iteration
2) 1) "c"

    Stream

    There is no direct support from Redis to stream Set structure data.