I have 3 arrays: array "words" of pairs ["id": "word"] by the length 5000000, array "ids" of unique ids by the length 13000 and array "dict" of unique words (dictionary) by the length 500000. This is my code:
matrix = sparse.lil_matrix((len(ids), len(dict)))
for i in words:
matrix[id.index(i['id']), dict.index(i['word'])] += 1.0
But it works too slow (I haven't got a matrix after 15 hours of work). Are there any ideas to optimize my code?
First of all don't name your array dict
, it is confusing as well as hides the built-in type dict
.
The problem here is that you're doing everything in quadratic time, so convert your arrays dict
and id
to a dictionary first where each word
or id
point to its index.
matrix = sparse.lil_matrix((len(ids), len(dict)))
dict_from_dict = {word: ind for ind, word in enumerate(dict)}
dict_from_id = {id: ind for ind, id in enumerate(id)}
for i in words:
matrix[dict_from_id[i['id']], dict_from_dict[i['word']] += 1.0