Search code examples
pythonredisredis-py

Redis pattern: how would you cache free / occupied ressource with expiration?

Here is the problem I was working on this week, and I am kind of hitting a wall here.

Let's say I have 100 resources available to do some quick task.

What I want to do is for the client, as fast a possible:

  1. fetch the first available ressource
  2. mark it as occupied
  3. use it
  4. mark it as free. For this kind of thing, I think the use of a sorted set is the best.

But because my client is not very safe and can sometimes fail in the middle of the code it runs I really want to set expiration time when I mark a resource as occupied so resources are can't in the occupied state forever.

It sounds like a very common problem and I'm sure there is a lot of literature out there on how to fix it with Redis but I could not find any.

I found many patterns and example for "Maintaining a global leaderboard" kind of problem, but none of those examples dealt with key expiration.

I currently have a solution like this:

for ressource in ressources:
    if GET <ressource> == 0:
        SET <ressource> 1, EX=10
        use_ressource(<ressource>)
        SET <ressource> 0, EX=10
    else:
        continue

Thing is, as soon as I have lots of resources used, this can take lots of operations to find the first free resource, and although Redis is really fast, this snippet does not scale well.

Solution

  • Off the top of my head:

    • Maintain a set of free resources
    • Maintain a set of used resources
    • Set up a keyspace listener on the expired event notification

    When a resource is needed, randomly select one with SRANDMEMBER and move it to the in-use resources set with SMOVE. In this same transaction, set up a simple expire key with a good prefix, the name/type of the resource, and required TTL with SETEX.

    Set up a redis keyspace notification consumer (still new, but check out their newest tech Redis Gears for a super simplified version of this!) that listens to the expired events for your assigned prefix. When one of these events occurs, run the same SMOVE logic above but just move the resource back into the free resources set.

    Regarding the actual resources themselves, when they finish, have them self-expire their tracking keys and the notification consumer can handle the state refreshing :)

    This should give you the flexibility you need!

    Similar question here, and some answers may be of use: How to "EXPIRE" the "HSET" child key in redis?