Search code examples
pythontensorflownlppytorchhuggingface-transformers

Which model/technique to use for specific sentence extraction?

I have a dataset of tens of thousands of dialogues / conversations between a customer and customer support. These dialogues, which could be forum posts, or long-winded email conversations, have been hand-annotated to highlight the sentence containing the customers problem. For example:

Dear agent, I am writing to you because I have a very annoying problem with my washing machine. I bought it three weeks ago and was very happy with it. However, this morning the door does not lock properly. Please help

Dear customer.... etc

The highlighted sentence would be:

However, this morning the door does not lock properly.

  1. What approaches can I take to model this, so that in future I can automatically extract the customers problem? The domain of the datasets are broad, but within the hardware space, so it could be appliances, gadgets, machinery etc.
  2. What is this type of problem called? I thought this might be called "intent recognition", but most guides seem to refer to multiclass classification. The sentence either is or isn't the customers problem. I considered analysing each sentence and performing binary classification, but I'd like to explore options that take into account the context of the rest of the conversation if possible.
  3. What resources are available to research how to implement this in Python (using tensorflow or pytorch)

I found a model on HuggingFace which has been pre-trained with customer dialogues, and have read the research paper, so I was considering fine-tuning this as a starting point, but I only have experience with text (multiclass/multilabel) classification when it comes to transformers.

Solution

  • If you want to get a specific sentence (without any modification) from the original input text, that is often referred to as 'span classification' where the output is the index of the first and last word of the specific sentence. The state-of-the-art now is the attention models like BERT .You can check the Bert models that are designed for the 'span classification' problem in huggingface as RobertaForQuestionAnswering https://huggingface.co/docs/transformers/model_doc/roberta#transformers.TFRobertaForQuestionAnswering that uses TensorFlow or PyTorch library.