Update: I found that I could not even pass a class in the enqueue
, is this possible to do this?
I want to pass a item
object (a model in Django) as the parameter in queue.enqueue(grabber.parse_body, item)
, but the rqworker
will raise this Exception (once I tried to pass the item as parameter in spite of whether I use it):
File "/usr/local/lib/python3.4/dist-packages/rq/job.py", line 53, in unpickle raise UnpickleError('Could not unpickle.', pickled_string, e) rq.exceptions.UnpickleError: ('Could not unpickle.', ImproperlyConfigured('Requested setting INSTALLED_APPS, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.',))
Seems that parse_body
don't know what the item
is?
I know I could just pass the items's id as parameter, and then deal with id in parse_body
, but is this possible to use item
directly in parse_body
?
My main code:
(enqueue.py
and grabber.py
are independent script, not in the Django apps.)
enqueue.py
import os
import grabber
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "website.settings.development")
queue = django_rq.get_queue('default')
need_to_parse = Post.objects.all()
for item in need_to_parse:
queue.enqueue(grabber.parse_body, item)
grabber.py
def parse_body(item):
print(item)
The problem is that RQ's default pickler is cPickle which does not know how to serialize django model instances. A simpler approach would be to use model_to_dict and pass a pickable object to your queue.
from django.models import model_to_dict
my_dict = model_to_dict(my_instance,fields=[],exclude=[])
If you are intent on using django model instances in your queue, you can create your own Job class that uses a PickleSerializer first and then set your Queue.job to your new CustomJob(Job). https://docs.djangoproject.com/en/1.7/topics/http/sessions/#bundled-serializers