Search code examples
pythondjangopython-rqdjango-rq

How to pass Non-essential Built-in object as parameter in django-rq enqueue


Update: I found that I could not even pass a class in the enqueue, is this possible to do this?


I want to pass a item object (a model in Django) as the parameter in queue.enqueue(grabber.parse_body, item), but the rqworker will raise this Exception (once I tried to pass the item as parameter in spite of whether I use it):

File "/usr/local/lib/python3.4/dist-packages/rq/job.py", line 53, in unpickle raise UnpickleError('Could not unpickle.', pickled_string, e) rq.exceptions.UnpickleError: ('Could not unpickle.', ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.',))

Seems that parse_body don't know what the item is?

I know I could just pass the items's id as parameter, and then deal with id in parse_body, but is this possible to use item directly in parse_body?


My main code: (enqueue.py and grabber.py are independent script, not in the Django apps.)

enqueue.py

import os
import grabber
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "website.settings.development")

queue = django_rq.get_queue('default')
need_to_parse = Post.objects.all()
    for item in need_to_parse:
        queue.enqueue(grabber.parse_body, item)

grabber.py

def parse_body(item):
    print(item)

Solution

  • The problem is that RQ's default pickler is cPickle which does not know how to serialize django model instances. A simpler approach would be to use model_to_dict and pass a pickable object to your queue.

    from django.models import model_to_dict
    my_dict = model_to_dict(my_instance,fields=[],exclude=[])
    

    If you are intent on using django model instances in your queue, you can create your own Job class that uses a PickleSerializer first and then set your Queue.job to your new CustomJob(Job). https://docs.djangoproject.com/en/1.7/topics/http/sessions/#bundled-serializers