Search code examples
djangobeanstalkd

Creating beanstalkd jobs on django model signals


I want to create beanstalkd jobs when a django model is saved. For this I am listening to the model's post_save signal and the associated handler is creating a beanstalkd job like below for example (in myapp/signals.py):

@receiver(post_save, sender=MyModel)
def create_job(sender, **kwargs):
    beanstalk = beanstalkc.Connection(
        host='example.com',
        port=11300
    )
    beanstalk.use('test')
    beanstalk.puts('job');

However I do not like the fact that I am creating a new connection each time I receive a signal. My questions are:

  1. Would that be good if I connected to beanstalkd outside the function, at the top of the module and reuse the connection instead?
  2. However If I reused the connection and the connection broke for some reasons, the entire process would suffer, right? So, what would be a better solution here?

Solution

  • Finally I ended up following this approach:

    I created a class inside common/util/beanstalkd_client.py:

    from django.conf import settings
    import beanstalkc
    
    
    class _BeanstalkdClient:
        def __init__(self):
            self.beanstalk = beanstalkc.Connection(
                host=settings.BEANSTALKD.get('host'),
                port=int(settings.BEANSTALKD.get('port'))
            )
    
            self.job = None
    
        def watch(self, tube, ignore_default=True):
            self.beanstalk.watch(tube)
    
            if ignore_default:
                self.beanstalk.ignore('default')
    
        def use(self, tube):
            self.beanstalk.use(tube)
    
        def reserve(self):
            self.job = self.beanstalk.reserve()
            return self.job
    
        def put(self, payload):
            self.beanstalk.put(payload)
    
    
    beanstalkd_client = _BeanstalkdClient()
    

    This allows me to reuse the beanstalkd_client and handles the connection inside. Since python will always create one instance of a module, beanstalkd_client module will always have one connection to beanstalkd.

    Then in my signals.py:

    from django.db.models.signals import post_save, post_delete
    from django.dispatch import receiver
    from django.conf import settings
    
    from common.util.beanstalkd_client import beanstalkd_client
    
    beanstalkd_client.use(settings.BEANSTALKD.get('my_tube_name'))
    
    
    @receiver(post_save, sender=Address)
    def es_create_index(sender, **kwargs):
        beanstalkd_client.put('my awesome job!')
    

    However this still does not handle the 2nd point: What if the connection to beanstalkd is broken!