I have a large file (~10 GB) which contains t = 1000 (n x n)-matrices where n = 1000. The data file contains float numbers. I wrote a program which prints every matrix as a .png image. The problem is, that it gets slower and slower. The first image needs less than a second to print. Image 350 needs more than 2 minutes! The plot below shows the problem for t = 27 matrices / images.
Here is my code I use. Does someone see why it is getting slower and slower?
import itertools as it
import matplotlib.pyplot as plt
import time
def printImage(M, k):
fig = plt.imshow(M, interpolation='nearest')
fig.set_cmap('hot')
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)
plt.axis('off')
plt.tight_layout()
plt.savefig(str(k) + '.png', bbox_inches='tight', pad_inches=0, dpi=300)
plt.ioff()
n = 1000
t = 1000
with open('data', 'r') as f:
for i in range(t):
t0 = time.clock()
try:
items = [list(map(float, i.split())) for i in it.islice(f, n)]
except:
raise
else:
printImage(items,i)
t1 = time.clock()
print(str(i) + '/' + str(t) + ' ' + str(t1-t0))
The issue is probably because you're not closing your figures, which consumes more and more memory.
Modify your plotting to read
import itertools as it
import matplotlib.pyplot as plt
import time
def printImage(M, k):
figobj = plt.figure()
fig = plt.imshow(M, interpolation='nearest')
fig.set_cmap('hot')
fig.axes.get_xaxis().set_visible(False)
fig.axes.get_yaxis().set_visible(False)
plt.axis('off')
plt.tight_layout()
plt.savefig(str(k) + '.png', bbox_inches='tight', pad_inches=0, dpi=300)
plt.close(figobj)
You probably don't need the call to plt.ioff
either.