fractionate a combination generator to distribute it with celery

Question

here is my line :

generator = chain((product('ABCDE', repeat=k) for k in range(1,10)))

and i would like to distribute it over several computer.

I'am going to use celery (but i'am open on other stuff).

but for the moment i have no clue on how to fraction this line to distribute the generation.

If anyone have any clue

thanks

Answer 1

Here's a simple way to produce 5 different generators:

from itertools import product, islice

letters = 'ABCDE'

def make_generator(first_letter):
    return ((first_letter,) + rest
            for k in range(9)
            for rest in product(letters, repeat=k))

for letter in letters:
    print map(''.join, islice(make_generator(letter), 20)), '...'

Output:

['A', 'AA', 'AB', 'AC', 'AD', 'AE', 'AAA', 'AAB', 'AAC', 'AAD', 'AAE', 'ABA', 'ABB', 'ABC', 'ABD', 'ABE', 'ACA', 'ACB', 'ACC', 'ACD'] ...
['B', 'BA', 'BB', 'BC', 'BD', 'BE', 'BAA', 'BAB', 'BAC', 'BAD', 'BAE', 'BBA', 'BBB', 'BBC', 'BBD', 'BBE', 'BCA', 'BCB', 'BCC', 'BCD'] ...
['C', 'CA', 'CB', 'CC', 'CD', 'CE', 'CAA', 'CAB', 'CAC', 'CAD', 'CAE', 'CBA', 'CBB', 'CBC', 'CBD', 'CBE', 'CCA', 'CCB', 'CCC', 'CCD'] ...
['D', 'DA', 'DB', 'DC', 'DD', 'DE', 'DAA', 'DAB', 'DAC', 'DAD', 'DAE', 'DBA', 'DBB', 'DBC', 'DBD', 'DBE', 'DCA', 'DCB', 'DCC', 'DCD'] ...
['E', 'EA', 'EB', 'EC', 'ED', 'EE', 'EAA', 'EAB', 'EAC', 'EAD', 'EAE', 'EBA', 'EBB', 'EBC', 'EBD', 'EBE', 'ECA', 'ECB', 'ECC', 'ECD'] ...

Similarly you can produce 25 by fixing the first two letters, etc.

Note that I don't use chain because it doesn't do anything when passed a single iterable. You might want chain.from_iterable but I doubt it because that will yield single letters.

make_generator can alternatively be written:

for k in range(9):
    for rest in product(letters, repeat=k):
        yield (first_letter,) + rest