Extracting Table Data (Perl Cookbook, 2nd Edition)

## 20.19. Extracting Table Data

### 20.19.1. Problem

You have data in an HTML table, and you would like to turn that into a Perl data structure. For example, you want to monitor changes to an author's CPAN module list.

### 20.19.2. Solution

Use the HTML::TableContentParser module from CPAN:

>     use HTML::TableContentParser;

>     $tcp = HTML::TableContentParser->new;
>     $tables = $tcp->parse($HTML);

>     foreach $table (@$tables) {
>       @headers = map { $_->{data} } @{ $table->{headers} };
>       # attributes of table tag available as keys in hash
>       $table_width = $table->{width};

>       foreach $row (@{ $tables->{rows} }) {
>         # attributes of tr tag available as keys in hash
>         foreach $col (@{ $row->{cols} }) {
>           # attributes of td tag available as keys in hash
>           $data = $col->{data};
>         }
>       }
>     }

### 20.19.3. Discussion

The HTML::TableContentParser module converts all tables in the HTML document into a Perl data structure. As with HTML tables, there are three layers of nesting in the data structure: the table, the row, and the data in that row.

Each table, row, and data tag is represented as a hash reference. The hash keys correspond to attributes of the tag that defined that table, row, or cell. In addition, the value for a special key gives the contents of the table, row, or cell. In a table, the value for the `rows` key is a reference to an array of rows. In a row, the `cols` key points to an array of cells. In a cell, the `data` key holds the HTML contents of the data tag.

For example, take the following table:

>     <table width="100%" bgcolor="#ffffff">
>       <tr>
>         <td>Larry &amp; Gloria</td>
>         <td>Mountain View</td>
>         <td>California</td>
>       </tr>
>       <tr>
>         <td><b>Tom</b></td>
>         <td>Boulder</td>
>         <td>Colorado</td>
>       </tr>
>       <tr>
>         <td>Nathan &amp; Jenine</td>
>         <td>Fort Collins</td>
>         <td>Colorado</td>
>       </tr>
>     </table>

The `parse` method returns this data structure:

>     [
>       {
>         'width' => '100%',
>         'bgcolor' => '#ffffff',
>         'rows' => [
>                    {
>                     'cells' => [
>                                 { 'data' => 'Larry &amp; Gloria' },
>                                 { 'data' => 'Mountain View' },
>                                 { 'data' => 'California' },
>                                ],
>                     'data' => "\n      "
>                    },
>                    {
>                     'cells' => [
>                                 { 'data' => '<b>Tom</b>' },
>                                 { 'data' => 'Boulder' },
>                                 { 'data' => 'Colorado' },
>                                ],
>                     'data' => "\n      "
>                    },
>                    {
>                     'cells' => [
>                                 { 'data' => 'Nathan &amp; Jenine' },
>                                 { 'data' => 'Fort Collins' },
>                                 { 'data' => 'Colorado' },
>                                ],
>                     'data' => "\n      "
>                    }
>                   ]
>       }
>     ]

The data tags still contain tags and entities. If you don't want the tags and entities, remove them by hand using techniques from Recipe 20.6.

Example 20-11 fetches a particular CPAN author's page and displays in plain text the modules they own. You could use this as part of a system that notifies you when your favorite CPAN authors do something new.

#### Example 20-11. Dump modules for a particular CPAN author

>       #!/usr/bin/perl -w
>       # dump-cpan-modules-for-author - display modules a CPAN author owns
>       use LWP::Simple;
>       use URI;
>       use HTML::TableContentParser;
>       use HTML::Entities;
>       use strict;
>       our $URL = shift || 'http://search.cpan.org/author/TOMC/';
>       my $tables = get_tables($URL);
>       my $modules = $tables->[4];    # 5th table holds module data
>       foreach my $r (@{ $modules->{rows} }) {
>         my ($module_name, $module_link, $status, $description) =
>             parse_module_row($r, $URL);
>         print "$module_name <$module_link>\n\t$status\n\t$description\n\n";
>       }
>       sub get_tables {
>         my $URL = shift;
>         my $page = get($URL);
>         my $tcp = new HTML::TableContentParser;
>         return $tcp->parse($page);
>       }
>       sub parse_module_row {
>         my ($row, $URL) = @_;
>         my ($module_html, $module_link, $module_name, $status, $description);
>         # extract cells
>         $module_html = $row->{cells}[0]{data};  # link and name in HTML
>         $status      = $row->{cells}[1]{data};  # status string and link
>         $description = $row->{cells}[2]{data};  # description only
>         $status =~ s{<.*?>}{  }g; # naive link removal, works on this simple HTML
>         # separate module link and name from html
>         ($module_link, $module_name) = $module_html =~ m{href="(.*?)".*?>(.*)<}i;
>         $module_link = URI->new_abs($module_link, $URL); # resolve relative links
>         # clean up entities and tags
>         decode_entities($module_name);
>         decode_entities($description);
>         return ($module_name, $module_link, $status, $description);
>       }

### 20.19.4. See Also

The documentation for the CPAN module HTML::TableContentParser; http://search.cpan.org

Copyright © 2003 O'Reilly & Associates. All rights reserved.