Search code examples
pythonmemory-managementpython-itertools

Python itertool variations, memory maximum reached


I am currently looking to generate a list of numbers with a specific number of digits, my code currently as follows:

| Python 2.7 |

import itertools

inp = raw_input('Number of digits to write?:')
inp = int(inp)
inp2 = raw_input('File name?:')
inp2 = inp2 + '.txt'
variants = ["".join(item) for item in itertools.product("0123456789", repeat=inp)]

variant = open(inp2, 'w')

for number in variants:
    variant.write("%s\n" % number)

As you can see i am trying to produce multiple different files with the output placed line by line for each new number.

I know that the list may be of issue, as it is storing all the possible numbers in memory within that list. My question is: With the digits being more then 7, there is a memory issue, how would I go about reducing the memory needed or put multiple files together to generate a list with the same type of data.

would a for loop work to in essence 'append' two lists together ( say for the 4 digit file and the 5 digit file to create essentially a 9 digit file ) without the use of this specific itertools implementation?

Perhaps some sort of recursion? ( I still do not understand how to write recursive functions etc. Im a noob when it comes to Programing in general)


Solution

  • just use the iterator as its intended to be used ... the whole point of iterators is to not store everything in memory at once ...

    variants = itertools.product("0123456789", repeat=inp)
    
    variant = open(inp2, 'w')
    
    for number in variants:
        variant.write("%s\n" % (" ".join(number))
    

    alternatively you could use a generator instead that would be functionally equivelent

    variants = ("".join(item) for item in itertools.product("0123456789", repeat=inp)) #now its a generator expression
    
    
    with open("outfile","wb") as variant:
        for number in variants:
            variant.write("%s\n"%number)
    

    as pointed out you could do this much easier as

    max_value = 10**n
    with open("outfile","wb") as variant:
        for number in xrange(max_value):
            variant.write("{0:09d}\n".format(number))