python loop increases its memory usage rapidly

Question

I have a python script that runs a loop. Within this loop, the function DoDebugInfo is called, once per loop iteration. This function basically prints some pictures to the hard disk using matplotlib, export a KML file and do some other calculations, and returns nothing.

I'm having the problem that python, for each run, the function DoDebugInfo eats more and more RAM. I guess some variable are increasing it's size on each loop.

I added the following lines before and after the call:

print '=== before: ' + str(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000)
DoDebugInfo(inputs)
print '=== after: ' + str(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss / 1000)

The output is:

=== before: 71598.08
=== after: 170237.952
=== before: 170237.952
=== after: 255696.896
=== before: 255696.896
=== after: 341409.792

As you can see, before the call the program has a memory footprint, and after the call it increases, but stays stable until before the next call.

why is this? since DoDebugInfo(inputs) is a function that returns nothing, how can it be that some variables stay on memory? is there a need to clear all variables at the end of the function?

Edit: the DoDebugInfo imports this functions:

def plot_line(x,y,kind,lab_x,lab_y,filename):
    fig = plt.figure(figsize=(11,6),dpi=300)
    ax = fig.add_subplot(111)
    ax.grid(True,which='both')
    #print 'plotting'
    if type(x[0]) is datetime.datetime:
        #print 'datetime detected'
        ax.plot_date(matplotlib.dates.date2num(x),y,kind)
        ax.fmt_xdata = DateFormatter('%H')
        ax.autoscale_view()
        fig.autofmt_xdate()
    else:   
        #print 'no datetime'
        ax.plot(x,y,kind)
    xlabel = ax.set_xlabel(lab_x)
    ax.set_ylabel(lab_y)
    fig.savefig(filename,bbox_extra_artists=[xlabel], bbox_inches='tight')

def plot_hist(x,Nbins,lab_x,lab_y,filename):
    fig = plt.figure(figsize=(11,6),dpi=300)
    ax = fig.add_subplot(111)
    ax.grid(True,which='both')
    ax.hist(x,Nbins)
    xlabel = ax.set_xlabel(lab_x)
    ax.set_ylabel(lab_y)
    fig.savefig(filename,bbox_extra_artists=[xlabel], bbox_inches='tight')

and plots 10 figures to the disk using something like:

plot_line(index,alt,'-','Drive Index','Altitude in m',output_dir + 'name.png')

if I comment the lines that use plot_line the problem does not happen, so the leak should be on this lines of code.

Thanks

Answer 1

The problem relies on so many figures being created and never closed. Somehow python keeps them all alive.

I added the line

plt.close()

to each of my plot functions plot_line and plot_hist and the problem is gone.