how to improve speed of django haystack search

Question

I want to create a search engine in my django environment for the simple data structure:

| id         | comapany name    |
|:-----------|-----------------:|
| 12345678   | company A's name |
| 12345687   | peoples pizza a/s|
| 87654321   | sub's for pugs   |

There will be about 800,000 companies and I only want to search by name. When the name is found the ID is returned in my django.

I've tried various set ups with haystack, whoosh and such but I keep getting really slow search results as I raise from my test data set of ~500 to the 800,000. The search some times takes almost an hour.

I'm using the Paas Heroku so I thought I would try an integrated paid service (searly's elasticsearch implementation). This helped, but as I arrive at about 80,000 companies it starts getting really slow again.

Installed Apps

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',

    # Added.
    'haystack',

    # Then your usual apps...
]

More settings.py

import os
from urlparse import urlparse

es = urlparse(os.environ.get('SEARCHBOX_URL') or 'http://127.0.0.1:9200/')

port = es.port or 80

HAYSTACK_CONNECTIONS = {
   'default': {
       'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
       'URL': es.scheme + '://' + es.hostname + ':' + str(port),
       'INDEX_NAME': 'documents',
   },


if es.username:
   HAYSTACK_CONNECTIONS['default']['KWARGS'] = {"http_auth": es.username + ':' + es.password}

search_indexes.py

from haystack import indexes

from hello.models import Article


class ArticleIndex(indexes.SearchIndex, indexes.Indexable):
    '''
    defines the model for the serach Engine.
    '''
    text = indexes.CharField(document=True, use_template=True)
    pub_date = indexes.DateTimeField(model_attr='pub_date')
    # pub_date line was commented out previously
    content_auto = indexes.EdgeNgramField(model_attr='title')

    def get_model(self):
        return Article

    def index_queryset(self, using=None):
        """Used when the entire index for model is updated."""
        return self.get_model().objects.all()

article_text.txt

{{ object.title }}
{{ object.user.get_full_name }}
{{ object.body }}

urls.py

url(r'^search/$', views.search_titles, name='search'),

views.py

def search_titles(request):
    txt = request.POST.get('search_text', '')
    if txt and len(txt) >= 4:
        articles = SearchQuerySet().autocomplete(content_auto=txt)
    # if the post request is empty, return nothing
    # this prevents internal server error with jquery
    else:
        articles = []
    return render_to_response('scripts/ajax_search.html',
                              {'articles': articles})

search.html

{% if articles.count > 0 %}
    <!-- simply prints the links to the cvr numbers-->
    <!-- for article in articles -->
    {% for article in "x"|rjust:"15" %} 
        <li><a href="{{ article.object.get_absolute_url }}">{{ article.object.title }}</a></li>
    {% endfor %}

{% else %}

    <li>Try again, or try CVR + &#x23ce;</li>

{% endif %}

index.html (where i call the search engine)

{% csrf_token %}
<input  type="text" id="search" name="search" />

<!-- This <ul> all company names end up-->
<ul id ="search-results"></ul>

Answer 1

I changed my ves.py search method to:

txt = request.POST.get('search_text', '')
articles = []
suggestedSearchTerm = ""
if txt and len(txt) >= 4:
    sqs = SearchQuerySet()
    sqs.query.set_limits(low=0, high=8)
    sqs = sqs.filter(content=txt)
    articles = sqs.query.get_results()
    suggestedSearchTerm = SearchQuerySet().spelling_suggestion(txt)
    if suggestedSearchTerm == txt:
        suggestedSearchTerm = ''
    else:
      suggestedSearchTerm = suggestedSearchTerm.lower()