Search code examples
pythondatabaselistmemorypython-itertools

Handling large list of lists in python


I have this mathematical task in which I am supposed to find some combinations, etc. That doesn't matter, the problem is that I am trying to do it with itertools module and it worked fine on smaller combinations (6 - places), but now I want to do the same for large combination (18 - places) so here I run into problem because I only have 8GB of RAM and this list comes around 5GB and with my system running it consumes all RAM and then program drops MemoryError. So my question is: what would be good alternative to the method I'm using(code below)?

poliedar_kom = list(itertools.combinations_with_replacement(range(0, 13), 18)) poliedar_len = len(poliedar_kom)

So when I have this list and it's length, the rest of program is going through every value in list and checking for condition with values in another smaller list. As I already said that's problem because this list gets too big for my PC, but I'm probably doing something wrong.

Note: I am using latest Python 3.8 64-bit

Summary: I have too big list of lists through which I have to loop to check values for conditions.

EDIT: I appreciate all answers, I have to try them now, if you have any new possible solution to the problem please post it.

EDIT 2: Thanks everyone, you helped me really much. I marked answer that pointed me to Youtube video because it made me realize that my code is already generator. Thanks everyone!!!


Solution

  • Use generators for large data ranges, time and space complexity of the code will not increase exponentially with large data size, refer to the link for more details:

    https://www.youtube.com/watch?v=bD05uGo_sVI