Research papers classification on the basis of title of the research paper

Question

Dear all I am working on a project in which I have to categories research papers into their appropriate fields using titles of papers. For example if a phrase "computer network" occurs somewhere in then title then this paper should be tagged as related to the concept "computer network". I have 3 million titles of research papers. So I want to know how I should start. I have tried to use tf-idf but could not get actual results. Does someone know about a library to do this task easily? Kindly suggest one. I shall be thankful.

Answer 1

If you don't know categories in advance, than it's not classification, but instead clustering. Basically, you need to do following:

1. Select algorithm.
2. Select and extract features.
3. Apply algorithm to features.

Quite simple. You only need to choose combination of algorithm and features that fits your case best.

When talking about clustering, there are several popular choices. K-means is considered one of the best and has enormous number of implementations, even in libraries not specialized in ML. Another popular choice is Expectation-Maximization (EM) algorithm. Both of them, however, require initial guess about number of classes. If you can't predict number of classes even approximately, other algorithms - such as hierarchical clustering or DBSCAN - may work for you better (see discussion here).

As for features, words themselves normally work fine for clustering by topic. Just tokenize your text, normalize and vectorize words (see this if you don't know what it all means).

Some useful links:

• Clustering text documents using k-means
• NLTK clustering package
• Statistical Machine Learning for Text Classification with scikit-learn and NLTK

Note: all links in this answer are about Python, since it has really powerful and convenient tools for this kind of tasks, but if you have another language of preference, you most probably will be able to find similar libraries for it too.