Search code examples
pythonnlpcorpusldagensim

Which gensim corpora class should I use to load an LDA transformed corpus? - Python

How do I load an LDA transformed corpus from python's gensim ? What i've tried:

from gensim import corpora, models
import numpy.random
numpy.random.seed(10)

doc0 = [(0, 1), (1, 1)]
doc1 = [(0,1)]
doc2 = [(0, 1), (1, 1)]
doc3 = [(0, 3), (1, 1)]

corpus = [doc0,doc1,doc2,doc3]
dictionary = corpora.Dictionary(corpus)

tfidf = models.TfidfModel(corpus)
corpus_tfidf = tfidf[corpus]
corpus_tfidf.save('x.corpus_tfidf')

# To access the tfidf fitted corpus i've saved i used corpora.MmCorpus.load()
corpus_tfidf = corpora.MmCorpus.load('x.corpus_tfidf')

lda = models.ldamodel.LdaModel(corpus_tfidf, id2word=dictionary, num_topics=2)
corpus_lda = lda[corpus]
corpus_lda.save('x.corpus_lda')

for i,j in enumerate(corpus_lda):
  print j, corpus[i]

The above code will output:

[(0, 0.54259038344543631), (1, 0.45740961655456358)] [(0, 1), (1, 1)]
[(0, 0.56718063124157458), (1, 0.43281936875842542)] [(0, 1)]
[(0, 0.54255407573666647), (1, 0.45744592426333358)] [(0, 1), (1, 1)]
[(0, 0.75229707773868093), (1, 0.2477029222613191)] [(0, 3), (1, 1)]

# [(<topic_number_from x.corpus_lda model>, 
#   <probability of this topic for this document>), 
#  (<topic# from lda model>, <prob of this top for this doc>)] [<document[i] from corpus>]

If i want to load the saved LDA transformed corpus, which class from gensim should i be using to load?

I have tried using corpora.MmCorpus.load(), it doesn't give me the same output of the transformed corpus as shown above:

>>> lda_corpus = corpora.MmCorpus.load('x.corpus_lda')
>>> for i,j in enumerate(lda_corpus):
...   print j, corpus[i]
... 
[(0, 0.55087839240547309), (1, 0.44912160759452685)] [(0, 1), (1, 1)]
[(0, 0.56715974584850259), (1, 0.43284025415149735)] [(0, 1)]
[(0, 0.54275680271070581), (1, 0.45724319728929413)] [(0, 1), (1, 1)]
[(0, 0.75233330695720912), (1, 0.24766669304279079)] [(0, 3), (1, 1)]
Solution

  • There are more problems in your code.

    To save a corpus in MatrixMarket format, you'd 

    corpora.MmCorpus.serialize('x.corpus_lda', corpus_lda)

    The docs are here.

    You're training on corpus_tfidf, but then transforming only lda[corpus] (no tfidf). Either use tfidf or plain bag-of-words, but use it consistently.