Say you have millions of Android GCM device keys and you want to send them in a management script. This script will take loads of time to finish as it's processing the keys in the DB as a queue.
Question: How do you implement this faster? how do you send these notifications in parallel? how do you get to near-real-time push notifications?
One solution would to to instantiate an X number of celery workers where each worker is responsible for an offset Y at which it starts fetching from MySQL. Example:
Worker 1: starts at offset 0,
Worker 2: starts at offset 10,000,
Worker 3: starts at offset 20,000,
Worker 4: starts at offset 30,000,
Worker 5: starts at offset 40,000,
Worker 1: Restarts at offset 50,000,
Worker 2: Restarts at offset 60,000,
... etc
Is this a viable solution?
Create list of tasks as a Celery group. Also because you have to retrieve all records from Android model it's good to create separate celery task which will do it in background:
@shared_task
def push_notification(offset, limit):
for android in Android.objects.all()[offset:offset+limit]:
pass
@shared_task
def push_notification_to_all():
count = Android.objects.all().count()
limit = 100
group(push_notification.s(offset, limit) for offset in range(0, count, limit)()
push_notification_to_all.delay()
Also instead of sending