How do I set up Haystack's QueuedSignalProcessor to use Redis?

Question

I have a project which I would like to migrate from using Haystack's queued_search library, using Redis as the Queue backend. Before this point, everything was working great with Haystack and Django, I'm just having trouble changing the signal processor.

My understanding is this actually requires several layers of code:

• Redis :: This is the data store
• RQ :: This is an implementation of Queues (https://code.google.com/p/queues/) using Redis
• django-rq :: This is a Django wrapper for RQ, where I can specify connection settings
• queued_search :: This is Haystack signal processing library. This looks for a generic QUEUE_BACKEND, which is set to 'rq'.

1) I have installed Redis locally. It appears to be running swimmingly:

> brew install redis
> redis-server /usr/local/etc/redis.conf
65024:M 24 Jul 17:08:31.779 * Increased maximum number of open files to 10032 (it was originally set to 256).
65024:M 24 Jul 17:08:31.780 # Server started, Redis version 3.0.2
65024:M 24 Jul 17:08:31.780 * The server is now ready to accept connections on port 6379

2) I have installed RQ, django-rq and queued_search

> pip install rq
> pip install django-rq
> pip install queued_search

3) In my settings.py file I have added queued_search and django_rq to the installed apps and changed the haystack signal processor setting:

INSTALLED_APPS = (
    'queued_search',
    'django_rq',
    'haystack',
    ....
  )

HAYSTACK_SIGNAL_PROCESSOR = 'queued_search.signals.QueuedSignalProcessor'

4) In my settings.py file, I have set the QUEUE_BACKEND to 'rq'. The queued_search application needs a QUEUE_BACKEND to be defined:

QUEUE_BACKEND = 'rq'

4) In my settings.py file I have added the RQ_QUEUES settings, with an index called 'haystack_search_queue' set up for queued_search:

RQ_QUEUES = {
    'haystack_search_queue': {
        'HOST': 'localhost',
        'PORT': 6379,
        'DB': 0
    }
}

5) I have added django_rq to urls.py:

urlpatterns += patterns('',
    (r'^django-rq/', include('django_rq.urls')),
)

But when I attempt to save an object, I keep getting this error in rq/connections.py line 70:

NoRedisConnectionException at /admin/app/model/pk/
Could not resolve a Redis connection

There are so many moving parts, I can't find any documentation on how to set this up end-to-end. Am I missing a step or perhaps have too many steps? Is there an easier way to set up the QueuedSignalProcessor?

Answer 1

I was able to set this up using a slightly different library configuration:

• Redis :: This is the data store
• RQ :: This is an implementation of Queues (https://code.google.com/p/queues/) using Redis
• django-rq :: This is a Django wrapper for RQ
• haystack-rqueue :: This is Haystack signal processing library that hands off the haystack signals to django-rq.

Switching to haystack-rqueue instead of queued_search allowed all the components to talk with one-another successfully!

There's an example implementation in haystack-rqueue repository: https://github.com/mandx/haystack-rqueue/tree/master/haystack_rq_test