recreate zdiffstore in Redis for Ruby

Question

I am trying to create a transaction similar to a Redis sdiffstore for a sorted set. I have a Redis "main" (temp:'..u_id) sorted set and a second "remove" set. I want to return the "main" sorted set with scores after removing any matching members in the "remove" set.

My structure has ~200 "main" keys with ~200 scored members in each. The "remove" set has ~50 members for each one of the main keys. For every request I need to loop through all ~200 users which is proving to be a costly process. Below is an portion of my existing server side script:

#Generate list of user ids 
local usercard = redis.call('smembers', 'temp:user')

--#Loop through all available users
--#Get array of users
for i=1,table.getn(usercard)-1,1 do 
    local u_id = usercard[i] 

    --#Create a copy of the user and store into temp:user_id
    redis.call('zunionstore', 'temp:..u_id,1,'user:'..u_id)

    --#remove unwanted
    redis.call('zrem','temp:'..u_id, unpack(redis.call('smembers','remove:'..u_id)))
end

The zunionstore and zrem combination is a very expensive command and the loop proves to be the limiting factor preventing the ability to scale. I know I am not making best use of the Redis thread as I cant scale past ~50 concurrent connections before everything bogs down.

As I am relatively new to Redis:

1) Is Redis the best db choice for challenge?

2) Is there a more efficient key structure?

3) Is there a more efficient way to structure the server side script or am I better off doing a pair of calls (zrange for the "main" sorted set and smembers for the "remove" set) to do the processing in Ruby given the big(O) will remain rather large?

Answer 1

I am not sure I understand your problem entirely. Do you really need to write data or just to get the filtered list on the application side? Do you always need that for all 200 users or just for one?

Also, you shouldn't subtract one in the loop, you are probably missing your last user.

If you need all the users and don't need to keep the results in Redis, you may consider something like this (untested):

local r = {}
local t, l, s, w
for _,u_id in ipairs(redis.call('smembers', 'temp:user')) do
  t = redis.call('zrange', 'user:' .. u_id, 0, -1, "withscores")
  l = redis.call('smembers', 'remove:' .. u_id)
  s = {}
  for _,x in ipairs(l) do s[x] = true end
  w = {}
  for i=1,#t,2 do
    if not s[t[i]] then w[t[i]] = w[t[i+1]] end
  end
  r[u_id] = w
end

return r