I am generating all combinations from a set of numbers, and then want to generate combinations of those combinations. Because of the vast number of possible combinations I keep getting a memory error. I have looked at the following questions but none have really solved my problem:
Creating all combinations of a set and running out of memory
Python itertools.combinations() memory problems
Python list memory error
I am generating my list using the following method:
#generate all combinations of 1 and 0 of size 30
set_1 = itertools.product([0,1], repeat = 30)
#generate all combinations of set 1, of size 5
set_2 = [tuple(c) for c in pulp.allcombinations(set_1, 5)]
for sets in set_2:
print(sets)
Them memory error occurs while it is generating set_2. I would like to still be able to iterate over set_2 as I will need to access the sets later on. I have considered writing the sets to a txt file, but I would like to save that as a last resort.
Instead of using a list comprehension which creates a list in your memory you can use a generator expression to store set2 and save your memory:
set_2 = (tuple(c) for c in pulp.allcombinations(set_1, 5))
Generators are like list comprehension except that they doesn't store the values in memory and just produce the values on demand.But they are one shot iterators and you can not iterate over them again like the result of a list comprehension.