Search code examples
pythondjangoreal-time

Realtime string matching results in Django

I'm not really sure where to get started on this, so really looking for some pointers or keywords to search for.

Let's say we have a file:

myHugeFile = open('someGiganticFile.txt')

And searching it takes, say 10 minutes from top to bottom, nothing is indexed. 

for line in myHugeFile:
    if 'keyword' in line:
        send_line_to_web_user(line)

I want to integrate this search into a django page but the page loading will probably time out for large files. The user would be able to search for 'keyword' and then a list on the page is populated in realtime in front of them with search results as they pop up.

Is there something existing to do this? That would save me making some uber-complex polling system for a buffer of results which exists for each user's session. Perhaps a different framework or library? Venture into node.js land? I'm a bit at loss as to what to search for apart from "realtime polling django" and so forth.

Solution

  • This is an ideal case for considering shifting to node.js, or some other asynchronous framework. Tornado is a good one if you want to stick to Python.

    Also, I remember a friend using whoosh as a search engine for an internship project. He was fairly satisfied with it, so I can recommend it.