Should the "perplexity" (or "score") go up or down in the LDA implementation of Scikit-learn?
I'd like to know what does the perplexity and score means in the LDA implementation of Scikit-learn. Those functions are obscure.
At the very least, I need to know if those values increase or decrease when the model is better. I've searched but it's somehow unclear. I feel that the perplexity should go down, but I'd like a clear answer on how those values should go up or down.
Perplexity is the measure of how well a model predicts a sample.
According to Latent Dirichlet Allocation by Blei, Ng, & Jordan,
[W]e computed the perplexity of a held-out test set to evaluate the models. The perplexity, used by convention in language modeling, is monotonically decreasing in the likelihood of the test data, and is algebraicly equivalent to the inverse of the geometric mean per-word likelihood. A lower perplexity score indicates better generalization performance.
This can be seen with the following graph in the paper:
In essense, since perplexity is equivalent to the inverse of the geometric mean, a lower perplexity implies data is more likely. As such, as the number of topics increase, the perplexity of the model should decrease.