Celery + gevent using only one CPU core

Question

I've issues with performance load running Celery with gevent, everything is running on the same core on my VPS.

Here's a screenshot of 4 Celery instance with 20 gevent concurrency each

How to fix this ? What am I doing wrong ?

Here's my first task :

def update_sender():
    items = models.Item.objects.filter(active=True).all()
    count = items.count()
    items = [i.id for i in items]
    step = count / settings.WORKERS
    for job in list(chunks(items, step)):
        update_item.apply_async(args=[job])

Calling the following sub task:

def update_item(items):
    for item in items:
        try:
            i = models.Item.objects.get(id=item)
            url = "someurl"
            rep = requests.get(url)
            jrep = rep.json()
            tracker = ItemTracker(i, jrep)
            if tracker.skip():
                continue
            if tracker.method1():
                if not tracker.method2():
                    tracker.method3()
                tracker.save()

It's all about doing a lot of HTTP requests and updating database concurrently.

Answer 1

Celery with gevent still only uses a single process, its just starting multiple greenlets inside of the process, but it is still only executing one greenlet at a time. To allow using more than 1 core, you need to start multiple celery processes using something like celery-multi