Search code examples
ruby-on-railsnlpnltk

What is the best way to do Natural Language Processing in Rails app?


I have a Rails app. I need to implement automatic text categorization algorithms and possibly more NLP capabilities in app. I believe Ruby does not have good NLP tools available as Python has. I am using a separate resque server for process background jobs. I believe I have following

  1. Run python scripts using resque jobs
  2. Run a flask application on a separate server which can either talk to resque job or can automatically update the app database with processed results.
  3. Use Ruby tools mentioned in this thread
  4. Any other suggestions welcome

Please let me know what is the best way to do it. Are there any similar working examples?


Solution

  • I had the same problem a few months ago. After a bit of research and testing this is the solution I implemented

    Run several python processes As many as one machine can hold. And use as many machines as you need.

    Use ZeroMQ to communicate between the web servers and the machines running python processes

    Do not use HTTP to communicate because the overhead is significant and it will be very slow compared to ZeroMQ. You will also not need an as complex handler with ZeroMQ as you would with HTTP

    Take care to expose ZeroMQ sockets to internal networks only, or you would need to set up authentication on each python server

    Another option is to just use one of the many available NLP APIs, if don't need any corpus based algorithms (such as POS tagging, Sentiment Analysis, etc).