I have a Rails app. I need to implement automatic text categorization algorithms and possibly more NLP capabilities in app. I believe Ruby does not have good NLP tools available as Python has. I am using a separate resque server for process background jobs. I believe I have following
Please let me know what is the best way to do it. Are there any similar working examples?
I had the same problem a few months ago. After a bit of research and testing this is the solution I implemented
Run several python processes As many as one machine can hold. And use as many machines as you need.
Use ZeroMQ to communicate between the web servers and the machines running python processes
Do not use HTTP to communicate because the overhead is significant and it will be very slow compared to ZeroMQ. You will also not need an as complex handler with ZeroMQ as you would with HTTP
Take care to expose ZeroMQ sockets to internal networks only, or you would need to set up authentication on each python server
Another option is to just use one of the many available NLP APIs, if don't need any corpus based algorithms (such as POS tagging, Sentiment Analysis, etc).