How to optimize memory usage of itertools.combinations?

Question

I want to generate all possible combinations for my missing letters of password.

I choose itertools.combinations over itertools.products because it produces about 4% of the later and it's fast but after running my code gets killed because of out of memory.

def genb(lst, word_length):
    with concurrent.futures.ThreadPoolExecutor(max_workers=multiprocessing.cpu_count()) as executor:
        comb = ("".join(combination) for combination in itertools.combinations(lst, word_length))
        for future in concurrent.futures.as_completed(executor.submit(combine, comb)):
            result = future.result()
            if result:
                return result
    return False

Is there a way to reduce memory usage of the above code? I have 256GB of RAM and 64 threads running.

Answer 1

This memory issue is not related to the combinations() call which uses only a tiny, fixed amount of memory.

Instead, the issue is caused by having too many instances of concurrent futures.

A possible solution is to create only a handful of futures (each running a separate core) and to submit batches of password attempts to evaluate.