I have a 30MB .txt file, with one line of data (30 Million Digit Number)
Unfortunately, every method I've tried (mmap.read()
, readline()
, allocating 1GB of RAM, for loops) takes 45+ minutes to completely read the file.
Every method I found on the internet seems to work on the fact that each line is small, therefore the memory consumption is only as big as the biggest line in the file. Here's the code I've been using.
start = time.clock()
z = open('Number.txt','r+')
m = mmap.mmap(z.fileno(), 0)
global a
a = int(m.read())
z.close()
end = time.clock()
secs = (end - start)
print("Number read in","%s" % (secs),"seconds.", file=f)
print("Number read in","%s" % (secs),"seconds.")
f.flush()
del end,start,secs,z,m
Other than splitting the number from one line to various lines; which I'd rather not do, is there a cleaner method which won't require the better part of an hour?
By the way, I don't necessarily have to use text files.
I have: Windows 8.1 64-Bit, 16GB RAM, Python 3.5.1
I used the gmpy2 module to convert the string to a number.
start = time.clock()
z=open('Number.txt','r+')
data=z.read()
global a
a=gmpy2.mpz(data)
end = time.clock()
secs = (end - start)
print("Number read in","%s" % (secs),"seconds.", file=f)
print("Number read in","%s" % (secs),"seconds.")
f.flush()
del end,secs,start,z,data
It worked in 3 seconds, much slower, but at least it gave me an integer value.
Thank you all for your invaluable answers, however I'm going to mark this one as soon as possible.