VMS-to-Unix Phrase Book

VMS-to-Unix Phrase Book

Return

3.8 Data Driven Scripts

Problem

You want to process an arbitrary list of files, the names of which are stored in a data file.

Solution

Expand the list of files on the command line, or expand the contents of the file into a shell variable.

Discussion

Let's say you want to create a script that can accept one or more filenames on the command line, or as an alternative, read those filenames from a data file.

Method One: do everything from the command line.

Here is our input data file, and two sample runs:

unix> cat file-list.data
file1 file2 file3 file4
file5
file6

file7

file8
file9 file10
file11
unix>
unix> ./file-list-1.sh myfile
1 of 1:         [myfile]
unix>
unix> ./file-list-1.sh `cat file-list.data`
1 of 11:        [file1]
2 of 11:        [file2]
3 of 11:        [file3]
4 of 11:        [file4]
5 of 11:        [file5]
6 of 11:        [file6]
7 of 11:        [file7]
8 of 11:        [file8]
9 of 11:        [file9]
10 of 11:       [file10]
11 of 11:       [file11]
unix>

And here's the script itself:

unix> cat file-list-1.sh
#! /bin/sh
#
#         File: file-list-1.sh
#     Abstract: Demo program; list all args to STDOUT and exit.
#

    total=$#
    count=0
    while [ $# -gt 0 ]
    do
        count=`expr $count + 1`
        echo "$count of $total: \t[$1]"
        shift
    done
unix>

The back tick operator is used to execute and replace the cat command with the contents of our data file.

In this program we also:

Remember the total number of arguments on the command line at the start of the program by saving the value of the special shell variable $# which contains the current number of arguments.
Perform a loop while there are still arguments to be processed. The syntax [ ] is actually a shorthand for the test command. See man test for more information about this program and expressions it can evaluate.
Increment our counter by using the expr command for evaluating expressions and returning the value as STDOUT. See man expr for more information.
Use the shift to discard the current argument. This makes what was the second argument ($2) now the first ($1) and changes the value of $#.
Also note the use of the meta-character \t in the echo command for inserting a tab character. See man echo for more information on the available meta-characters.

Notice that in the input file, sometimes we put more than one filename on a line, sometimes exactly one filename per line, and even included some blank lines. But our program still worked because the shell automatically regularized the data for us as a series of blank delimited words.

Method Two: expand data file from within script.

Here's another, slightly different example. Let's say that for validity checking, it is important that our script take an exact number of parameters. In this case we are either looking for a single filename, or the special keyword 'ALL'. Assuming the same input data, here is a sample execution:

unix> ./file-list-2.sh myfile
./file-list-2.sh: too few args, expecting 3
unix> ./file-list-2.sh first second myfile
     first arg: first
    second arg: second

         db: [myfile]
unix>
unix> ./file-list-2.sh first second ALL
     first arg: first
    second arg: second

         db: [file1]
         db: [file2]
         db: [file3]
         db: [file4]
         db: [file5]
         db: [file6]
         db: [file7]
         db: [file8]
         db: [file9]
         db: [file10]
         db: [file11]
unix>

And here is the script.

unix> cat file-list-2.sh
#! /bin/sh
#
#         File: file-list-2.sh
#     Abstract: Demo program; list all args to STDOUT and exit.
#

    if [ $# -gt 3 ]
    then echo "$0: too many args after '$3'" >&2
         exit 2
    elif [ $# -lt 3 ]
    then echo "$0: too few args, expecting 3" >&2
         exit 2
    fi;

    echo "     first arg: $1"
    echo "    second arg: $2"
    echo

    if [ "$3" = "ALL" ]
    then inputs=`cat file-list.data`
    else inputs="$3"
    fi;

    for db in $inputs
    do
       echo "\t db: [$db]"
    done


#==[ EOF: file-list-2.sh ]==
unix>

Here's how it works:

We test the number of parameters using $# and issue an error message to STDERR (>&2) if there are too many or not enough. Remember, however, that an empty string, as in VMS, is a legal parameter and is expressed in the same manner. Thus the command:
```
    unix> ./file-list-2.sh first "" myfile
```
would be accepted by our script. This mechanism is designed to allow the satisfaction of a positional, but optional, parameter. If your application requires that a parameter have a non-blank value, you'll need to explicitly test for it.
We test the third parameter ($3) for our special keyword. If found, we load a shell variable with the contents of our file list data file, using the same technique (back ticks) as in our previous example for expanding the contents of the file on the command line.
The shell for-loop construct can be used to iterate over a set of space delimited values. The first parameter after the keyword for is the name of the shell variable we want to be assigned each iterative value. Notice that we do not say $db since this would dereference the variable db and place its value on the command line instead. In other words, the value of the dereferenced variable would be used as the for-loop variable. The keyword in acts as a delimiter followed by the list of values. Here we do want the shell variable inputs dereferenced so that it is replaced with a list of zero or more words. The shell will then iterate over each of these words in turn.

Bonus Tip

But for-loops aren't just for breakfast (or shell scripts) anymore. They also can come in handy interactively:

     for file in *.bak; do clear; head $file; rm -i $file; done

In otherwords, for every backup file in the current directory (*.bak), clear the terminal screen (clear), display the first 10 lines of the next file (head $file), ask if it should be deleted (rm -i $file). Lather, rinse, repeat.