Extracting URLs (Perl Cookbook, 2nd Edition)

## 20.3. Extracting URLs

### 20.3.1. Problem

You want to extract all URLs from an HTML file. For example, you have downloaded a page that lists the MP3 files downloadable from some site. You want to extract those MP3s' URLS so you can filter the list and write a program to download the ones you want.

### 20.3.2. Solution

Use the HTML::LinkExtor module from CPAN:

>     use HTML::LinkExtor;

>     $parser = HTML::LinkExtor->new(undef, $base_url);
>     $parser->parse_file($filename);
>     @links = $parser->links;
>     foreach $linkarray (@links) {
>         my @element = @$linkarray;
>         my $elt_type = shift @element;                  # element type

>         # possibly test whether this is an element we're interested in
>         while (@element) {
>             # extract the next attribute and its value
>             my ($attr_name, $attr_value) = splice(@element, 0, 2);
>             # ... do something with them ...
>         }
>     }

### 20.3.3. Discussion

You can use HTML::LinkExtor in two different ways: either by calling `links` to get a list of all links in the document once it is completely parsed, or by passing a code reference in the first argument to `new`. The referenced function is called on each link as the document is parsed.

The `links` method clears the link list, so call it only once per parsed document. It returns a reference to an array of elements. Each element is itself an array reference with an HTML::Element object at the front followed by a list of attribute name and attribute value pairs. For instance, the HTML:

>     <A HREF="http://www.perl.com/">Home page</A>
>     <IMG SRC="images/big.gif" LOWSRC="images/big-lowres.gif">

would return a data structure like this:

>     [
>       [ a,   href   => "http://www.perl.com/" ],
>       [ img, src    => "images/big.gif",
>              lowsrc => "images/big-lowres.gif" ]
>     ]

Here's an example of how to use `$elt_type` and `$attr_name` to print out and anchor an image:

>     if ($elt_type eq 'a' && $attr_name eq 'href') {
>         print "ANCHOR: $attr_value\n"
>             if $attr_value->scheme =~ /http|ftp/;
>     }
>     if ($elt_type eq 'img' && $attr_name eq 'src') {
>         print "IMAGE:  $attr_value\n";
>     }

To extract links only to MP3 files, you'd say:

>     foreach my $linkarray (@links) {
>          my ($elt_type, %attrs) = @$linkarray;

>          if ($elt_type eq 'a' && $attrs{'href'} =~ /\.mp3$/i) {
>            # do something with $attr{'href'}, the URL of the mp3 file
>          }
>      }

Example 20-2 is a complete program that takes as its arguments a URL, such as file:///tmp/testing.html or http://www.ora.com/, and produces on standard output an alphabetically sorted list of unique URLs linked from that site.

#### Example 20-2. xurl

>       #!/usr/bin/perl -w
>       # xurl - extract unique, sorted list of links from URL
>       use HTML::LinkExtor;
>       use LWP::Simple;

>       $base_url = shift;
>       $parser = HTML::LinkExtor->new(undef, $base_url);
>       $parser->parse(get($base_url))->eof;
>       @links = $parser->links;
>       foreach $linkarray (@links) {
>           my @element  = @$linkarray;
>           my $elt_type = shift @element;
>           while (@element) {
>               my ($attr_name , $attr_value) = splice(@element, 0, 2);
>               $seen{$attr_value}++;
>           }
>       }
>       for (sort keys %seen) { print $_, "\n" }

This program does have a limitation: if the `get` of `$base_url` involves a redirection, links resolve using the original URL instead of the URL after the redirection. To fix this, fetch the document with LWP::UserAgent and examine the response code to find out whether a redirection occurred. Once you know the post-redirection URL (if any), construct the HTML::LinkExtor object accordingly.

Here's an example of the run:

>     % xurl http://www.perl.com/CPAN
>     ftp://ftp@ftp.perl.com/CPAN/CPAN.html
>     http://language.perl.com/misc/CPAN.cgi
>     http://language.perl.com/misc/cpan_module
>     http://language.perl.com/misc/getcpan
>     http://www.perl.com/index.html
>     http://www.perl.com/gifs/lcb.xbm

In mail or Usenet messages, you may see URLs written as:

>     <URL:http://www.perl.com>

This is supposed to make it easy to pick URLs from messages:

>     @URLs = ($message =~ /<URL:(.*?)>/g);

### 20.3.4. See Also

The documentation for the CPAN modules LWP::Simple, HTML::LinkExtor, and HTML::Entities; Recipe 20.1

Copyright © 2003 O'Reilly & Associates. All rights reserved.