Writing a Unix-Style Filter Program (Perl Cookbook, 2nd Edition)

## 7.14. Writing a Unix-Style Filter Program

### 7.14.1. Problem

You want to write a program that takes a list of filenames on the command line and reads from `STDIN` if no filenames were given. You'd like the user to be able to give the file "`-`" to indicate `STDIN` or "`someprogram` `|`" to indicate the output of another program. You might want your program to modify the files in place or to produce output based on its input.

### 7.14.2. Solution

Read lines with `<>`:

>     while (<>) {
>         # do something with the line
>     }

### 7.14.3. Discussion

When you say:

>     while (<>) {
>         # ...
>      }

Perl translates this into:[14]

> [14]Except that the code written here won't work, because `ARGV` has internal magic.

>     unshift(@ARGV, "-") unless @ARGV;
>     while ($ARGV = shift @ARGV) {
>         unless (open(ARGV, $ARGV)) {
>             warn "Can't open $ARGV: $!\n";
>             next;
>         }
>         while (defined($_ = <ARGV>)) {
>             # ...
>         }
>     }

You can access `ARGV` and `$ARGV` inside the loop to read more from the filehandle or to find the filename currently being processed. Let's look at how this works.

### 7.14.3.1. Behavior

If the user supplies no arguments, Perl sets `@ARGV` to a single string, "`-`". This is shorthand for `STDIN` when opened for reading and `STDOUT` when opened for writing. It's also what lets the user of your program specify "`-`" as a filename on the command line to read from `STDIN`.

Next, the file-processing loop removes one argument at a time from `@ARGV` and copies the filename into the global variable `$ARGV`. If the file cannot be opened, Perl goes on to the next one. Otherwise, it processes a line at a time. When the file runs out, the loop goes back and opens the next one, repeating the process until `@ARGV` is exhausted.

The `open` statement didn't say `open(ARGV, "<", $ARGV)`. There's no extra less-than sign supplied. This allows for interesting effects, like passing the string "`gzip` `-dc` `file.gz` `|`" as an argument, to make your program read the output of the command "`gzip` `-dc` `file.gz`". See Recipe 16.6 for more about this use of magic open.

You can change `@ARGV` before or inside the loop. Let's say you don't want the default behavior of reading from `STDIN` if there aren't any arguments—you want it to default to all C or C++ source and header files. Insert this line before you start processing `<ARGV>`:

>     @ARGV = glob("*.[Cch]") unless @ARGV;

Process options before the loop, either with one of the Getopt libraries described in Chapter 15 or manually:

>     # arg demo 1: Process optional -c flag
>     if (@ARGV && $ARGV[0] eq "-c") {
>         $chop_first++;
>         shift;
>     }

>     # arg demo 2: Process optional -NUMBER flag
>     if (@ARGV && $ARGV[0] =~ /^-(\d+)$/) {
>         $columns = $1;
>         shift;
>     }

>     # arg demo 3: Process clustering -a, -i, -n, or -u flags
>     while (@ARGV && $ARGV[0] =~ /^-(.+)/ && (shift, ($_ = $1), 1)) {
>         next if /^$/;
>         s/a// && (++$append,      redo);
>         die "usage: $0 [-ainu] [filenames] ...\n";
>     }

Other than its implicit looping over command-line arguments, `<>` is not special. The special variables controlling I/O still apply; see Chapter 8 for more on them. You can set `$/` to set the line terminator, and `$`. contains the current line (record) number. If you undefine `$/`, you don't get the concatenated contents of all files at once; you get one complete file each time:

>     undef $/;
>     while (<>) {
>         # $_ now has the complete contents of
>         # the file whose name is in $ARGV
>     }

If you localize `$/`, the old value is automatically restored when the enclosing block exits:

>     {     # create block for local
>         local $/;         # record separator now undef
>         while (<>) {
>             # do something; called functions still have
>             # undeffed version of $/
>         }
>     }                             # $/ restored here

Because processing `<ARGV>` never explicitly closes filehandles, the record number in `$`. is not reset. If you don't like that, you can explicitly close the file yourself to reset `$`.:

>     while (<>) {
>         print "$ARGV:$.:$_";
>         close ARGV if eof;
>     }

The `eof` function defaults to checking the end-of-file status of the last file read. Since the last handle read was `ARGV`, `eof` reports whether we're at the end of the current file. If so, we close it and reset the `$`. variable. On the other hand, the special notation `eof( )` with parentheses but no argument checks if we've reached the end of all files in the `<ARGV>` processing.

### 7.14.3.2. Command-line options

Perl has command-line options, -n, -p, -a, and -i, to make writing filters and one-liners easier.

The -n option adds the `while` `(<>)` loop around your program text. It's normally used for filters like grep or programs that summarize the data they read. The program is shown in Example 7-2.

#### Example 7-2. findlogin1

>       #!/usr/bin/perl
>       # findlogin1 - print all lines containing the string "login"
>       while (<>) {# loop over files on command line
>           print if /login/;
>       }

The program in Example 7-2 could be written as shown in Example 7-3.

#### Example 7-3. findlogin2

>       #!/usr/bin/perl -n
>       # findlogin2 - print all lines containing the string "login"
>       print if /login/;

You can combine the -n and -e options to run Perl code from the command line:

>     % perl -ne 'print if /login/'

The -p option is like -n but adds a `print` right before the end of the loop. It's normally used for programs that translate their input, such as the program shown in Example 7-4.

#### Example 7-4. lowercase1

>       #!/usr/bin/perl
>       # lowercase - turn all lines into lowercase
>       while (<>) {                  # loop over lines on command line
>           s/(\p{Letter})/\l$1/g;    # change all letters to lowercase
>           print;
>       }

The program in Example 7-4 could be written as shown in Example 7-5.

#### Example 7-5. lowercase2

>       #!/usr/bin/perl -p
>       # lowercase - turn all lines into lowercase
>       s/(\p{Letter})/\l$1/g;# change all letters to lowercase

Or it could be written from the command line as:

>     % perl -pe 's/(\p{Letter})/\l$1/g'

While using -n or -p for implicit input looping, the special label `LINE`: is silently created for the whole input loop. That means that from an inner loop, you can skip to the following input record by using `next` `LINE` (which is like awk's `next` statement), or go on to the next file by closing `ARGV` (which is like awk's `nextfile` statement). This is shown in Example 7-6.

#### Example 7-6. countchunks

>       #!/usr/bin/perl -n
>       # countchunks - count how many words are used.
>       # skip comments, and bail on file if _ _END_ _
>       # or _ _DATA_ _ seen.
>       for (split /\W+/) {
>           next LINE if /^#/;
>           close ARGV if /_ _(DATA|END)_ _/;
>           $chunks++;
>       }
>       END { print "Found $chunks chunks\n" }

The tcsh keeps a .history file in a format such that every other line contains a commented out timestamp in Epoch seconds:

>     #+0894382237
>     less /etc/motd
>     #+0894382239
>     vi ~/.exrc
>     #+0894382242
>     date
>     #+0894382242
>     who
>     #+0894382288
>     telnet home

A simple one-liner can render that legible:

>     % perl -pe 's/^#\+(\d+)\n/localtime($1) . " "/e'
>     Tue May  5 09:30:37 1998     less /etc/motd
>     Tue May  5 09:30:39 1998     vi ~/.exrc
>     Tue May  5 09:30:42 1998     date
>     Tue May  5 09:30:42 1998     who
>     Tue May  5 09:31:28 1998     telnet home

The -i option changes each file on the command line. It is described in Recipe 7.16, and is normally used in conjunction with -p.

### 7.14.4. See Also

perlrun(1), and the "Switches" section of Chapter 19 of Programming Perl; Recipe 7.16; Recipe 16.6

Copyright © 2003 O'Reilly & Associates. All rights reserved.