Search code examples
perlmemory-leakslwp

Perl LWP memory leak?

I have a continuously running crawl (using Perl LWP) where I do extraction on crawled webpages using HTML::TreeBuilder. I assiduously delete every instance of HTML::TreeBuilder that I create, yet the memory footprint of the crawler rises slowly but steadily (I'm watching it tick up by 10 to 30 bytes with every page that is crawled (I'm using Valgrind to measure heap usage).

Right now the flow is something like:

  • Crawl page
  • Create new tree builder object
  • Parse page
  • Delete tree builder object

Does anyone have hints about whether there's a memory leak in either LWP or TreeBuilder? I've read a bunch of reports that LWP had memory leaks in it, but I haven't seen a response on that matter.

I'm running Perl v5.10.1

thanks!

Solution

  • I suppose that problem with HTML::TreeBuilder.

    Could you may to check following links:

    1) Stackoverflow memory leak in HTML::TreeBuilder

    2) Perlmonks Memory Leak? i'm clueless.

    I hope these links will help you.