How to avoid race conditions on celery tasks?

Question

Considering the following task:

@app.task(ignore_result=True)
def withdraw(user, requested_amount):
    if user.balance >= requested_amount:
        send_money(requested_amount)
        user.balance -= requested_amount
        user.save()

If this task gets executed twice, at the same time, it would result in an user with a negative balance... how can I solve it? It is just an example of race, but there are lots of situations like this in my code..

Answer 1

You can use this Celery cookbook recipe to implement a Lock which will make sure that only one task is run at a time, and then perhaps implement retry logic to then try the second task again later. Something like this;

def import_feed(self, feed_url):
    # The cache key consists of the task name and the MD5 digest
    # of the feed URL.
    feed_url_hexdigest = md5(feed_url).hexdigest()
    lock_id = '{0}-lock-{1}'.format(self.name, feed_url_hexdigest)
    logger.debug('Importing feed: %s', feed_url)
    with memcache_lock(lock_id, self.app.oid) as acquired:
        if acquired:
            return Feed.objects.import_feed(feed_url).url
    self.retry(countdown = 2)