I have three documents in a df:
id author document
12X john the cat sat
12Y jane the dog ran
12Z jane the hippo ate
These documents are converted into a corpus of TaggedDocuments with the tags being the typical practice of semantically meaningless ints:
def read_corpus(documents):
for i, plot in enumerate(documents):
yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(plot, max_len=30), [i])
train_corpus = list(read_corpus(df.document))
This corpus is then used to train my Doc2Vec model:
model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=55)
model.build_vocab(train_corpus)
model.train(train_corpus, total_examples=model.corpus_count, epochs=model.epochs)
The resulting vectors of the model are accessed like this:
model.docvecs.vectors_docs
How would I tie the original df to the resulting vectors? Now that all the documents are trained and vectors are identified for each one, I want to query the set of vectors by author. For example, if I want to return a set of vectors only for Jane, how would I do so?
I think the basic idea is to identify the int tags that correspond to Jane and then do something like this to access them:
from operator import itemgetter
a = model.docvecs.vectors_docs
b = [1, 2]
itemgetter(*b)(a)
How would I identify the tags though? They are only meaningful to the model and the tagged documents, so they don't join back to my original df.
I tried a simple example using Gensim. I think the approach here should work for you
import gensim
training_sentences = ['This is some document from Author {}'.format(i) for i in range(1,10)]
def read_corpus():
for i,line in enumerate(training_sentences):
# lets use the tag to identify the document and author
yield gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(line), ['Doc{}_Author{}'.format(i,i)])
You could also directly prepare the training corpus from pandas_df like shown below
data_df = pd.DataFrame({'doc':training_sentences,'doc_id':[i for i in range (1,10)],'author_id':[10+i for i in range (1,10)]})
data_df.head()
tagged_docs = data_df.apply(lambda x:gensim.models.doc2vec.TaggedDocument(gensim.utils.simple_preprocess(x.doc),['doc{}_auth{}'.format(x.doc_id,x.author_id)]),axis=1)
training_corpus = tagged_docs.values
>> array([ TaggedDocument(words=['this', 'is', 'some', 'document', 'from', 'author'], tags=['doc1_auth11']),
TaggedDocument(words=['this', 'is', 'some', 'document', 'from', 'author'], tags=['doc2_auth12']),
# training
model = gensim.models.doc2vec.Doc2Vec(vector_size=50, min_count=2, epochs=55)
train_corpus = list(read_corpus())
model.build_vocab(train_corpus)
model.train(train_corpus, total_examples=model.corpus_count, epochs=model.epochs)
# indexing
model.docvecs.index2entity
>>
['Doc0_Author0',
'Doc1_Author1',
'Doc2_Author2',
'Doc3_Author3',
'Doc4_Author4',
'Doc5_Author5',
'Doc6_Author6',
'Doc7_Author7',
'Doc8_Author8']
Now, to access the vector corresponding to document1 of author1, you could do
model.docvecs[model.docvecs.index2entity.index('Doc1_Author1')]
array([ 8.08026362e-03, 4.27437993e-03, -7.73820514e-03,
-7.40669528e-03, 6.36066869e-03, 4.03292105e-03,
9.60215740e-03, -4.26750770e-03, -1.34797185e-03,
-9.02472902e-03, 6.25275355e-03, -2.49505695e-03,
3.18572600e-03, 2.56929174e-03, -4.17032139e-03,
-2.33384431e-03, -5.10744564e-03, -5.29057207e-03,
5.41675789e-03, 5.83767192e-03, -5.91145828e-03,
5.91885624e-03, -1.00465110e-02, 8.32535885e-03,
9.72494949e-03, -7.35746371e-03, -1.86231872e-03,
8.94813929e-05, -4.11528209e-03, -9.72509012e-03,
-6.52212929e-03, -8.83922912e-03, 9.46981460e-03,
-3.90578934e-04, 6.74136635e-03, -5.24599617e-03,
9.73031297e-03, -8.77021812e-03, -5.55411633e-03,
-7.21857697e-03, -4.50362219e-03, -4.06361837e-03,
2.57276138e-03, 1.76626759e-06, -8.08755495e-03,
-1.48400548e-03, -5.26673114e-03, -7.78301107e-03,
-4.24248137e-04, -7.99000356e-03], dtype=float32)
yes this uses doc-author pair ordering, you could just use doc_id alone and maintain a separate index like {doc_id:author_id} in a python dict, if you want to filter by author then use {author_id : [docids,...]}