Search code examples
pythonnode.jsnlpchatbot

Building your own NLP API

I'm building a chatbot and I'm new to NLP.

(api.ai & AlchemyAPI are too expensive for my use case. And wit.ai seems to be buggy and constantly changing at the moment.)

For the NLP experts, how easily can I replicate their services locally?

My vision so far (with node, but open to Python):

  • entity extraction via StanfordNER
  • intent via NodeNatural's LogisticRegressionClassifier
  • training UI with text and validate/invalidate buttons (any prebuilt tools for this?)

Are entities and intents all I'll need for a chatbot? How good will NodeNatural/StanfordNER be compared to NLP-as-a-service? What headaches am I not seeing?

Solution

  • You seems to have done your home work. Like you said,following things will help you along the way,

    • any classification algorithm for identifying the intent such as sklearn's LinearSVC or LogisticRegression Classifier
    • Any good NER tool such as StanfordNER or CRFsuite. CRFsuite has easy to use pthon wrapping called pycrfsuite.
    • A sentiment analysis tool for more human like conversations. its better if you go with python because python has lot of free libraries for the same.

    Only benefit of services such as wit.ai or api.ai is their heavily trained ready to use intents and models.You too will be able to achive similar accuracy if you are able to provide decent amount of training to your bot.

    It's better if you build on some existing opensource libraries rather than building everything from scratch. Please check my opensource project on github for wit.ai/api.ai similar interface. Happy coding!