Search code examples
pythonpython-itertoolscartesian-product

Too big itertools.product in Python


i have a script as following:

set=[0,1]
sub=prod(set,repeat=10)
subset = [item for item in sub]

In there, subset contains 10 elements of combinations of set.

When i change repeat from 10 to 500, it is impossible to see subset! How can i deal with that problem?

I guess it becomes RAM invader!


Solution

  • That's not a subset you're making, that's just making a list of all the values from the product. Asking for all of them at once is equivalent to running a counter from 0 to 2 ** 500, which, in case it's not clear, would take roughly "heat death of the universe" time even on low-level, close to the metal languages. Written out, you'd have 3273390607896141870013189696827599152216642046043064789483291368096133796404674554883270092325904157150886684127560071009217256545885393053328527589376 different outputs.

    You cannot realize all the values, period, let alone all at once. If you want to see some of them, you can iterate the product and print as you go:

    valset = [0, 1]  # Don't name a variable set, it shadows the built-in set
    
    for subset in prod(valset, repeat=10):
        print(subset)  # Be prepared to Ctrl-C this, it will never end on its own
    

    or use itertools.islice to pull off a manageable number to listify:

    first10000subsets = list(itertools.islice(prod(valset, repeat=10), 10000))