Search code examples
pythonlistproductpython-itertools

How to use itertools product with a huge data


I would like to make a list of 81-tuple by using three elements, namely 1,2,3 in python.

I tried to find a solution, and then I found these useful links:

How to use itertools to compute all combinations with repeating elements?

and

Which itertools generator doesn't skip any combinations?

According to the above links, I should do the following

import itertools

list = []
for p in itertools.product(range(1, 3 + 1), repeat=81):
    list.append(p)

print(list)

But, my computer hangs. I think there is too much data in the list.

I want to know whether there is a command that prints only first 100-elements in list or the 101th to 200th in the list.


Solution

  • You can use itertools.islice:

    p = itertools.product(range(1, 3 + 1), repeat=81)
    s = itertools.islice(p, 101, 200)
    print(list(s))
    

    This will, however, iterate through all the elements until it reaches the starting index of the slice. So for ranges towards the end of an iterator with a gazillion elements (yours has 3**81 = 443426488243037769948249630619149892803 or in other words: too many to process let alone store), this will run into similar problems.

    For those later ranges, you would have to calculate the n-th element by hand and generate successors from there... See How to select specific item from cartesian product without calculating every other item for some inspiration.