I tried creating a simple Doc2Vec model:
sentences = []
sentences.append(doc2vec.TaggedDocument(words=[u'scarpe', u'rosse', u'con', u'tacco'], tags=[1]))
sentences.append(doc2vec.TaggedDocument(words=[u'scarpe', u'blu'], tags=[2]))
sentences.append(doc2vec.TaggedDocument(words=[u'scarponcini', u'Emporio', u'Armani'], tags=[3]))
sentences.append(doc2vec.TaggedDocument(words=[u'scarpe', u'marca', u'italiana'], tags=[4]))
sentences.append(doc2vec.TaggedDocument(words=[u'scarpe', u'bianche', u'senza', u'tacco'], tags=[5]))
model = Doc2Vec(alpha=0.025, min_alpha=0.025) # use fixed learning rate
model.build_vocab(sentences)
But I end up with an empty vocabulary. With some debugging I saw that inside the build_vocab() function a dictionary is actually created by the vocabulary.scan_vocab() function, but it's being deleted by the following vocabulary.prepare_vocab() function. More deeply, this is the function that causes the problem:
def keep_vocab_item(word, count, min_count, trim_rule=None):
"""Check that should we keep `word` in vocab or remove.
Parameters
----------
word : str
Input word.
count : int
Number of times that word contains in corpus.
min_count : int
Frequency threshold for `word`.
trim_rule : function, optional
Function for trimming entities from vocab, default behaviour is `vocab[w] <= min_reduce`.
Returns
-------
bool
True if `word` should stay, False otherwise.
"""
default_res = count >= min_count
if trim_rule is None:
return default_res # <-- ALWAYS RETURNS FALSE
else:
rule_res = trim_rule(word, count, min_count)
if rule_res == RULE_KEEP:
return True
elif rule_res == RULE_DISCARD:
return False
else:
return default_res
Does somebody understand the problem?
I found the answer myself, the default value for min_count is 5 and I had no words with a counter of 5 or more. I just had to change this line of code:
model = Doc2Vec(min_count=0, alpha=0.025, min_alpha=0.025)