Subdivide itertools.permutations process in Python

Question

Is there any way to subdivide a process that takes a lot of memory (itertools.permutations in this case) in order to improve the efficiency and to not run out of memory?

Answer 1

You've backed yourself into a corner with itertools.permutations. Do you really need to check all of the possible permutations? Your dictionary is pretty small, so just iterate over the dictionary itself and validate each word:

import itertools
from collections import Counter

with open('/usr/share/dict/american-english', 'r') as handle:
    dictionary = frozenset(line.strip() for line in handle)

def existing_words(letters, length):
    letters_counter = Counter(letters)
    letters_set = frozenset(letters)

    for word in dictionary:
        if len(word) != length:
            continue

        if set(word) <= letters_set and Counter(word) <= letters_counter:
            yield word

if __name__ == '__main__':
    for word in existing_words('abcdefghijklmnopqrst', 5):
        print word