Celery is sending 600K requests in less than two weeks

Question

I received the monthly bill today for Amazon SQS and I was surprised to see 600.000 requests usage to my queue.

All I am doing is running one single task every minute. How does this add up to 600.000 requests?

@celery.task(name='tasks.check_for_events')
@periodic_task(run_every=timedelta(minutes=1))  
def check_for_events():    
    now = datetime.utcnow().replace(tzinfo=utc,second=00, microsecond=00)
    events = Event.objects.filter(is_reminder_sent = False).filter(reminder_date_time__range=(now - timedelta(minutes=1), now))    
    dthandler = lambda obj: obj.isoformat() if isinstance(obj, datetime) else None    
    for event in events:
          sendEmail.delay( ...)


@celery.task(name='tasks.sendEmail')
def sendEmail(...)
    ....

I am still new to celery so it might be that I am doing something fundamentally wrong. Any tips please?

Answer 1

First, SQS considers sends, receives and deletes as requests. So that's

>>> 600000 / 3
200000

tasks.

Last month there were 31 days. So with the check for events task alone there were

>>> 60 * 24 * 31
44640

tasks or

>>> 44640 * 3
133920

requests used upfront.

Now, you'd only have to average

>>> (200000.0 - 44640) / 44640
3.4802867383512543

events per check_for_events invocation to get to 600k requests.

Now I'm not certain what volume of events you're dealing with, but perhaps this puts the usage more into perspective.