Search code examples
perlhtml-parsinghtml-tree

HTML::TagFilter remove div based on class

I'm trying to use a perl script to pull content from static html files on a server. I'd like to pull the content of a specific div. I know the div by its class name ("getme"). I can get to the div using HTML::TreeBuilder->look_down. How can I remove the div tag and get to just the content within it?

Example HTML

<body>
<div class="getme">
    <h2>Some Header</h2>
    <div class="another"><p>More text</p></div>
    <div class="yetanother">text text text</div>
</div>
<div class="second">...</div>
</body>

Perl so far

use strict;
use warnings;
use HTML::TreeBuilder;
use HTML::TagFilter;

my $unique_filename = '/path/to/saved/files/extracted_divs/' . get_timestamp();
my $guid_counter = 0;
my $field_sep = "|";

open FILEOUT, ">>", $unique_filename or die $!;

print FILEOUT "guid|published|url|title|body\n";
foreach my $file_name (@ARGV) {
    my $tree = HTML::TreeBuilder->new;
    my $filter = HTML::TagFilter->new(deny => { div => {class => ["getme"]} });
    $tree->parse_file($file_name);
    for my $subtree ($tree->look_down(_tag => "div", class => "getme")) {
        #my $html = $filter->filter($subtree->as_HTML);
        my $html = $subtree->as_HTML;
        #steamline HTML
        $html =~ s/(?<!\n)\z/\n/;
        #echo file name to screen so we know something is happening
        print $file_name . "\n";

        #replace slashes with spaces
        my $file_url = $file_name;
        $file_name =~ s/\//-/g;
        #remove ".html"
        $file_name =~ s/.html//g;

        #echo info to output file
        print FILEOUT $guid_counter++ . $field_sep . time . $field_sep;
        print FILEOUT $file_url . $field_sep . $file_name . $field_sep;
        print FILEOUT $html;
    }
    $tree = $tree->delete;
}
close (FILEOUT);

The filter just removes the class attribute. Can a rule be made to remove the whole tag, or is there a better approach to this?

Solution 
  • use Web::Query qw();
join '', Web::Query->new_from_html($html)->find('.getme > *')->html

    returns the string

    <h2>Some Header</h2><div class="another"><p>More text</div><div class="yetanother">text text text</div>