Mirroring Web Pages (Perl Cookbook, 2nd Edition)

## 20.10. Mirroring Web Pages

### 20.10.1. Problem

You want a local copy of a web page kept up-to-date.

### 20.10.2. Solution

Use LWP::Simple's `mirror` function:

>     use LWP::Simple;
>     mirror($URL, $local_filename);

### 20.10.3. Discussion

Although closely related to the `get` function discussed in Recipe 20.1, the `mirror` function doesn't download the file unconditionally. It adds the `If-Modified-Since` header to the GET request it creates, so the server does not transfer the file unless the file has been updated.

The `mirror` function mirrors only a single page, not a full tree. To mirror a set of pages, use this recipe in conjunction with Recipe 20.3. A good solution to mirroring an entire directory hierarchy can be found in the w3mir program, also found on CPAN, and the wget program from ftp.gnu.org.

Be careful! It's possible (and easy) to write programs that run amok and begin downloading all web pages on the net. This is not only poor etiquette, it's also an infinite task, since some pages are dynamically generated. It could also get you into trouble with someone who doesn't want their pages downloaded en masse.

### 20.10.4. See Also

The documentation for the CPAN module LWP::Simple; the HTTP specification at http://www.w3.org/pub/WWW/Protocols/HTTP/

Copyright © 2003 O'Reilly & Associates. All rights reserved.