Search code examples
xmlperlxml-sitemap

How do I remove duplicate notes from an XML document in Perl?


I have a sitemap video file xml with duplicated nodes :

<?xml version="1.0"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1"> 
<url>
<loc>http://www.tubtun.com/video/Samsung_42Channel_Wireless_SoundStand</loc>
<video:video>
    <video:title>Samsung 42Channel Wireless SoundStand</video:title>
    <video:description>Samsung 4.2Channel Wireless SoundStand</video:description>
    <video:thumbnail_loc>http://www.tubtun.com/media/files_thumbnail/user91/pl_5364844b0dc.jpg</video:thumbnail_loc>
    <video:player_loc>http://www.tubtun.com/modules/vPlayer/vPlayer.swf?f=http://www.tubtun.com/modules/vPlayer/vPlayercfg.php?fid=844b0dc2c7258f4de11</video:player_loc>
    <video:publication_date>2015-01-27</video:publication_date>
</video:video>
</url>
<url>
<loc>http://www.tubtun.com/video/Samsung_42Channel_Wireless_SoundStand</loc>
<video:video>
    <video:title>Samsung 42Channel Wireless SoundStand</video:title>
    <video:description>Samsung 4.2Channel Wireless SoundStand</video:description>
    <video:thumbnail_loc>http://www.tubtun.com/media/files_thumbnail/user91/pl_5364844b0dc.jpg</video:thumbnail_loc>
    <video:player_loc>http://www.tubtun.com/modules/vPlayer/vPlayer.swf?f=http://www.tubtun.com/modules/vPlayer/vPlayercfg.php?fid=844b0dc2c7258f4de11</video:player_loc>
    <video:publication_date>2015-01-27</video:publication_date>
</video:video>
</url>
.....

I have written a perl script to remove this duplicated data:

use strict;
use warnings;
use XML::LibXML;

my $file = 'sitemap.xml';
my $doc = XML::LibXML->load_xml( location => $file );

my %seen;
foreach my $uni ( $doc->findnodes('//url') ) {  # 'university' nodes only

    my $name = $uni->find('video:title');

    print "'$name' duplicated\n",
      $uni->unbindNode() if $seen{$name}++;  # Remove if seen before
}

$doc->toFile('clarified.xml'); # Print to file

Unfortunately, the file "clarified.xml" is the same as sitemap.xml.

I don't know what is wrong with my script.


Solution

  • I have it working, here's the code & I tried the solution provided in https://stackoverflow.com/a/4817929/235961

    use strict;
    use warnings;
    use XML::LibXML;
    
    my $file = 'sitemap.xml';
    my $doc = XML::LibXML->load_xml( location => $file );
    
    my %seen;
    foreach my $uni ( $doc->findnodes("//*[name() ='url']") ) {  # 'university' nodes only
    
        my $name = $uni->find('//video:title');
        print "'$name' duplicated\n",
          $uni->unbindNode() if $seen{$name}++;  # Remove if seen before
    }
    
    $doc->toFile('clarified.xml'); # Print to file