CONTENTS

Chapter 28. Saving Time on the Command Line

28.1 What's Special About the Unix Command Line

Summary Box

One of Unix's best features is the shell's command line. Why? Nearly every modern operating system has a command line; we don't use card readers with obscure job setup cards any more. What makes Unix's special?

The Unix shell command line allows lots of shortcuts. Some of these you'll find in other operating systems; some you won't. In this chapter, we'll introduce a lot of these shortcuts. Among other things, we'll discuss:

  • How to run commands more than onceSection 28.8).

  • Filename completion (Section 28.6, Section 28.7), which allows you to type the beginning of a filename and let the shell fill in the rest. (This is finally possible on certain Redmond-born OSes as well, but it usually involves a registry hack or two.)

  • Command substitution (Section 28.14), which lets you use the output from one command as arguments to another. (Note that this is different from pipelining.)

  • Process substitution in bash, and a script named ! for other shells, lets you put the output of a command into a temporary file and give that filename to a process.

  • The ability to repeat commands with various methods (Section 28.10, Section 28.11).

  • Handling of command lines that become too long (Section 28.17).

Some fundamental command-line features that we aren't discussing in this chapter, but which are discussed elsewhere, are:

You don't need to be a command-line virtuoso to use Unix effectively. But you'd be surprised at how much you can do with a few tricks. If all you can do at the command line is type ls or start Mozilla or the Gimp, you're missing out on a lot.

— ML

28.2 Reprinting Your Command Line with CTRL-r

You're logged in from home, running a program and answering a prompt. As you're almost done, modem noise prints xDxD@! on your screen. Where were you? Or you're typing a long command line and a friend interrupts you with write (Section 1.21) to say it's time for lunch. Do you have to press CTRL-u and start typing all over again?

If your system understands the rprnt character (usually set to CTRL-r), you can ask for the command line to be reprinted as it was. In fact, you can use CTRL-r any time you want to know what the system thinks you've typed on the current line — not just when you're interrupted. But this only works in the normal cooked input mode; programs like vi that do their own input processing may treat CTRL-r differently. Here's an example: 

% egrep '(10394|29433|49401)' /work/symtower/

Message from alison@ruby on ttyp2 at 12:02 ...
how about lunch?
EOF
CTRL-r
egrep '(10394|29433|49401)' /work/symtower/logs/*

After the interruption, I just pressed CTRL-r. It reprinted the stuff I'd started typing. I finished typing and pressed RETURN to run it.

If you use a shell like the Korn shell that has interactive command editing, you can probably use it to reprint the command line, too. In bash and other commands that use the readline file, though, from vi editing mode, CTRL-r still seems to start an Emacs-style reverse search. So I added this fix to my ~/.inputrc file: 

set editing-mode vi

# By default, in vi text-input mode, ^R does Emacs "reverse-i-search".
# In command mode, you can use the vi command ^L to redraw the line.
# Fix it in text-input mode:
"\C-r": redraw-current-line

— JP

28.3 Use Wildcards to Create Files?

The shells' [ ] (square bracket) wildcards will match a range of files. For instance, if you have files named afile, bfile, cfile, and dfile, you can print the first three by typing: 

% lpr [a-c]file

Now, let's say that you want to create some more files called efile, ffile, gfile, and hfile. What's wrong with typing the command line below? Try it. Instead of vi, you can use your favorite editor or the touch (Section 14.8) command: 

% vi [e-h]file     Doesn't make those four files
% ls 
afile   bfile   cfile   dfile

Stumped? Take a look at Section 1.13 about wildcard matching.

The answer: wildcards can't match names that don't exist yet. That's especially true with a command like touch ?file (Section 14.8) or touch *file — think how many filenames those wildcards could possibly create!

Section 28.4 explains shell { } operators that solve this problem. And, by the way, if you just created one new file named [e-h]file, simply quote (Section 27.12) its name to remove it: 

rm "[e-h]file"

— JP

28.4 Build Strings with { }

I've been finding more and more uses for the {} pattern-expansion characters in csh , tcsh, zsh, and bash. They're similar to *, ?, and [ ] (Section 33.2), but they don't match filenames the way that *, ?, and [ ] do. You can give them arbitrary text (not just filenames) to expand — that "expand-anything" ability is what makes them so useful.

Here are some examples to get you thinking:

In bash, the complete-into-braces editor command (which is bound to the M-{ key sequence by default in Emacs mode) expands a string into a list of matching filenames in braces. For example: 

$ ls pr*
prog1.c   prog2.c   program1.c   program2.c
$ cc pr META{
$ cc pr{og1.c,og2.c,ogram1.c,orgram2.c}

Then you can edit the brace expression.

— JP

28.5 String Editing (Colon) Operators

When the C shells, zsh, and bash do history substitutions (Section 30.8) they can also edit the substitution. The C shells and zsh — but not bash — can also edit variable substitutions (Section 35.9). (bash has a different syntax, which zsh understands, too.) For instance, in the first example below, when !$ contains /a/b/c, adding the "head" operator :h will give just the head of the pathname, /a/b.

For a complete but very terse list of these operators, see the csh manual page. We hope the examples below will help you understand these useful operators.

[Wait, Dan, what about & on the right-hand side to repeat the previous substitution? And there's more since Dan wrote this article (in 1983!). tcsh also has :u to convert the first lowercase letter to uppercase and :l to convert the first uppercase letter to lowercase. In zsh, :u converts all letters to uppercase and :l converts all letter to lowercase. zsh also has f and F to repeat a substitution until it fails — and even more. Check your shell's manual page. — JP]

— DR

28.6 Automatic Completion

If you hate typing long filenames, hostnames, command names — or almost anything on a command line — you should know about the shells' "completion" feature.

The basics are pretty simple: just press (in most shells) the TAB key, and the shell should "do the right thing." But how the shell decides what's "right" can be complicated — especially in newer shells, and especially in the latest zsh, which has incredibly customizable completion. As an example, when you press TAB in bash, the shell tries to complete a shell variable if the word begins with $, a username if the word begins with ~, a hostname if the word begins with @, or a command (including aliases and functions). If none of these works, bash finally tries filename completion. As another example, the original Korn shell does only simple filename completion, but the public domain Korn shell has more features.

On more-sophisticated shells, completion is actually a function of the shell's built-in customizable command editor. For instance, in tcsh, the TAB key is bound to (in other words, it runs) the editor's complete-word command. This key binding can be changed. And tcsh, like other recent shells, has plenty of other completion-related editor commands.

bash allows for the customization of the different types of completions, as well; you can define a file containing the hostnames to check (in /etc/hosts format) when the shell is asked to complete a hostname. Just set the environment variable HOSTFILE to the name of the file you want. There are extensive built-in functions in bash, each associated with a key, to allow for extremely flexible management of completions.

As you can see, completion varies shell to shell, so we'll give an overview here. For more details, see your shell's manpage.

28.6.1 General Example: Filename Completion

Let's look at an example of one type of completion, filename completion. Other types of completion work in generally the same way.

Filename completion is one of the most common types. You can type the initial part of a filename and then press the TAB key. (In the C shell, first enable completion by setting the variable filec (Section 30.9) or complete, then press ESC.) If the shell can figure out the complete filename from the part that you've typed, it will fill in the rest of the name. If not, it will fill in as much of the name as is unambiguous and then let you type some more. For example: 

$ ls 
alpha.c    alpha.o   beta.c
$ cc b  TAB
$ cc beta.c         Shell fills in the filename automatically

(With tcsh and csh, your terminal will beep if more than one file matches the name you've typed. If all this beeping drives you crazy, you can set the nobeep shell variable to turn it off.) In this case, only one filename begins with b, so the shell can fill in the entire name. This works with pathnames (Section 1.16) too: each time you press TAB, the shell completes the name up to the next slash (/) if it can.

If you type part of a filename and then type CTRL-d (in bash, type TAB twice), the shell lists all the files that match whatever you've typed. It then redisplays your command line and lets you continue typing. For example: 

% cc a CTRL-d
alpha.c    alpha.o
% cc alpha.

Two files begin with the letter "a"; the shell lists them. It then redisplays the cc command, letting you finish the filename.

Also, be forewarned that filename completion doesn't always work correctly. For example, you can't use filename completion within some older shell applications. You can't mix filename completion with wildcards in any shell except zsh. We won't go into detail about these rough edges, but if you're aware that they exist, you won't have trouble.

That last example shows a problem with filename completion: it's matching the ".o file," (Section 1.12) named alpha.o. It's a type of file that most users wouldn't want to manipulate from the command line; they'd rather the shell ignore all .o files. Section 28.7 explains the fignore list; it solves this problem in most cases. Section 31.10 shows an interesting shortcut to filename completion: cding to a directory by typing its "initials."

28.6.2 Menu Completion

The filename completion section showed how completion works by default: press TAB, and the shell completes as much as it can and then waits for you either to press TAB again (to see all possible completions) or to type enough of the word to make it unambigious.

Menu completion, supported by zsh with the -Y option, works differently. The name might be confusing at first: it doesn't "complete a menu," and it also doesn't pop up a menu of possible completions. Instead, menu completion replaces the word to be completed with a single match from the list of possible completions. Each time you press TAB again, the shell shows you the next possible match, in turn, under your cursor. If you like one of the choices, just keep typing the rest of the command line (or press ENTER to execute it). When the shell has shown all the possible matches, it rings the bell and restores the original text without a match.

Menu completion doesn't work just with filenames. If your shell supports it, menu completion probably works with all completion modes (filenames, hostnames, etc.).

28.6.3 Command-Specific Completion

tcsh and zsh let you customize completion even farther: specific completion instructions for each Unix command you define. For instance, the mail command wants email addresses on its command line, and you can declare a list of addresses that are available to complete (this could be a list of friends and associates you send a lot of mail to). You might use the ssh and telnet commands (Section 1.21) to connect to particular remote hosts, and you'd like to be able to complete the hostnames for those particular hosts. (The bash hostname completion feature reads hostnames from a file like /etc/hosts — but it only completes hostnames if the string starts with an @ character or if you use a special editor command for completing hostnames.)

The tcsh command complete defines these custom completions. The syntax is hairy, so I won't try to explain all of it here. Instead, let's look at an overall example from the MH email system (Section 6.2). You use MH commands directly from a shell prompt instead of first starting an email command interpreter and entering commands at the interpreter's own prompt, as you do with most other email packages. Most MH programs accept a mail folder name as one of their command-line arguments. A mail folder name starts with a + (plus sign)[1] and can appear anywhere in a command line.

MH mail folders can be stored anywhere on the filesystem — even on a networked filesystem on a remote computer. Here are the four lines that I put in my .tcshrc setup file (Section 3.3):

{ } Section 28.4

# Set up MH folder name completion for "folder", "refile", "scan", "show":
folders -fast -recurse | \
     sed -e '/DELETE$/d' -e 's/^/+/' > $HOME/Mail/folderlist
complete {folder,refile,scan,show} 'C@*@`cat $HOME/Mail/folderlist`@'

The first command builds a file named folderlist with a list of strings (in this case, folder names) to complete. I don't want completion to include folder names I'll never look in, so I filtered the folder output with sed (Section 34.1) to exclude the names I don't want — in this case, folder names ending with DELETE. (This list is also useful in other places, it turns out, not just in tcsh completion.) A + is prepended to each folder name because folders doesn't add the plus signs, but we need them for tcsh matching. So the first few lines of folderlist look like this: 

+drafts
+inbox
+jobs
+jobs/bay-area
+jobs/miscellaneous
   ...

The second command, complete, starts with a list in braces of the commands that should complete folder names. The next argument is complex and has lots of possible variations; this one matches any pattern included with backquotes (Section 28.14) from the cat (Section 12.2) command, which gives us the contents of folderlist. There are lots of variations! The bottom line is how this works... here's an example of completing a folder name: 

tcsh> scan +j TAB
tcsh> scan +jobs/m TAB
tcsh> scan +jobs/miscellaneous last:20

After completing the folder name (in two steps), tcsh leaves a space; I type the rest of the command line and press ENTER to run it.

28.6.4 Editor Functions for Completion

Some shells have customizable, built-in command-line editors that use key bindings to control how and where completion takes place. For example, in tcsh, pressing TAB invokes the complete-word function, but you can change TAB to do menu completion (as explained above) by binding the editor function complete-word-fwd to TAB key.

In bash, TAB does basic completion with the editor's complete function. But the bash editor has many more bindings than tcsh does. For instance, typing M-/ runs complete-filename, which treats the text before the cursor as a filename and does filename completion on it. Typing M-$ runs complete-variable, which treats the text before the cursor as a shell variable and does variable completion on it. There are plenty of variations — like C-x $, which invokes the possible-variable-completions function to list all shell variable names that could be completed. Section 28.4 has an example of M-{, the curly-brace completion function.

For details on your particular shell, check its manual page.

—JP, ML, and SJC

28.7 Don't Match Useless Files in Filename Completion

The shell variable fignore in csh and zsh (FIGNORE in bash and also zsh) lets you tell the shell that you aren't interested in some files when using filename completion (Section 28.6). For example, you may be more likely to refer to C language source files (whose names end with .c) than object files (.o files); you often need to edit your source files, while you may never need to look at your object modules. Set fignore to the suffixes that you want to ignore. For example, to ignore .o files in tcsh and csh, type:

set Section 35.9

% set fignore=(.o)

Once you've done this, file completion will ignore your .o files when you press the TAB key (ESC in csh) — unless a .o file is the only match it can find.

Most likely, there's a whole list of suffixes that you don't care about: .o (object modules), .out (random executables), .gz (gzipped files), ~ (Emacs backup files (Section 19.4)), and so on. Section 1.12 has a list of them. Here's how to set fignore to a list of filenames:[2] 

% set fignore=(.o .out .gz \~)     ...tcsh, csh, zsh
$ FIGNORE='.o:.out:.gz:~'          ...bash, zsh

fignore has no effect when you press CTRL-d to get a listing of the files that match in csh and tcsh. Those shells always give you a complete list of all possible completions.

—ML and JP

28.8 Repeating Commands

Let's start with some obvious ways to run commands more than once:

Whether each of those methods will work depends on the shell you're using and whether you have copy-and-paste built into your interface. All of those methods force you to take some action before each command line repeats — pressing the up-arrow key, for instance. That lets you control exactly when each command runs.

The next four articles show automated ways to repeat a command a certain number of times. You can "mix and match" some parts of different articles — the tips on read and sleep, for instance. Each article follows on to the one before, so we suggest glancing through all of them:

Finally, remember that you aren't stuck with the login shell you chose. If you want a feature that your shell doesn't have, you can use another shell temporarily by typing its name (like csh), running the commands you need, then typing exit to go back to your original shell.

— JP

28.9 Repeating and Varying Commands

28.9.1 A foreach Loop

When some people need to repeat a command on several files, the first thing they think of is command line editing (Section 30.14) or — as we show here — history substitution (Section 30.5):

-v Section 12.4, less Section 12.3

% cat -t -v /usr/fran/report | less
   ...
% ^fran/report^rob/file3
cat -t -v /usr/rob/file3 | less
   ...
% ^3^21
cat -t -v /usr/rob/file21 | less
   ...
%

The second substitution (changing 3 to 21) was quick to do, but the first one was longer. If there are lots of related commands like this, it can be easier to list all the variations at once — then let the shell do the dirty work. To do that, use the shell's foreach loop in C-type shells — or, in Bourne-type shells, use a for loop, shown later in this article. (zsh has both foreach and for loops.) You give the loop a list of the words that will change each time the command line is run. In this example, it's a list of filenames. The loop will step through the words, one by one, storing a word into a shell variable (Section 35.9), then running the command(s). The loop goes on until it has read all the words. For example: 

% foreach file (/usr/fran/report /usr/rob/file3 /usr/rob/file21) 
? cat -t -v $file | less 
? end 
    ...Shell runs cat -t -v /usr/fran/report | less...
    ...Shell runs cat -t -v /usr/rob/file3 | less...
    ...Shell runs cat -t -v /usr/rob/file21 | less...
%

The question marks (?) are secondary prompts (Section 28.12); the shell will keep printing them until you type the command end. Then the loop runs.

The list between the parentheses doesn't have to be filenames. Among other things, you can use wildcards (Section 1.13), backquotes (Section 28.14) (command substitution), variables (Section 35.9, Section 35.3), and the handy curly brace ({}) operators (Section 28.4). For example, you could have typed the above loop this way: 

% foreach file (/usr/fran/report /usr/rob/file{3,21})
? cat -t -v $file | less
? end

If you want the loop to stop before or after running each command, add the C shell operator $<. It reads keyboard input and waits for a RETURN. In this case, you can probably ignore the input; you'll use $< to make the loop wait. For example, to make the previous loop prompt before each command line:

set Section 35.9

% foreach file (/usr/fran/report /usr/rob/file{3,21}) 
? echo -n "Press RETURN to see $file--" 
? set x="$<" 
? cat -t -v $file | less 
? end 
Press RETURN to see /usr/fran/report--RETURN
    Shell runs cat -t -v /usr/fran/report | less...
Press RETURN to see /usr/rob/file3--RETURN
    Shell runs cat -t -v /usr/rob/file3 | less...
Press RETURN to see /usr/rob/file21--RETURN
    Shell runs cat -t -v /usr/rob/file21 | less...

The loop parameters don't need to be filenames. For instance, you could send a personalized email (Section 1.21) message to five people this way:[3]

cat - Section 12.2 

% foreach person (John Cathy Agnes Brett Elma)
? echo "Dear $person," | cat - formletter | mail $person
? end

The first line of the first letter will be "Dear John,"; the second letter "Dear Cathy,"; and so on.

Want to take this idea further? It's a part of shell programming (Section 35.2). I usually don't recommend shell programming with the C shell, but this is a handy technique to use interactively.

28.9.2 A for Loop

The for loop in Bourne-type shells is like the foreach loop shown earlier: it loops through a list of words, running one or more commands for each word in the list. This saves time when you want to run the same series of commands separately on several files.

Let's repeat an earlier example:

$ for file in /usr/fran/report /usr/rob/file2 /usr/rob/file3 
> do 
> cat -t -v $file | less 
> done 
    ...Shell runs cat -t -v /usr/fran/report | less...
    ...Shell runs cat -t -v /usr/rob/file2 | less...
    ...Shell runs cat -t -v /usr/rob/file3 | less...
$

The greater-than signs (>) are secondary prompts (Section 28.12); the Bourne shell will keep printing them until you type the command done. Then it runs the loop. You don't have to press RETURN after the do; you can type the first command on the same line after it.

In a shell script, the loop body (the lines between do and done) is usually indented for clarity.

The list after the in doesn't have to be filenames. Among other things, you can use backquotes (Section 28.14) (command substitution), variables (Section 35.9, Section 35.3), wildcards (Section 33.1), and, on shells like bash that have them, curly brace ({}) operators (Section 28.4). For example, you could have typed the previous loop this way: 

$ for file in /usr/fran/report /usr/rob/file[23]
> do cat -t -v $file | less
> done

If you want the loop to stop before or after running each command, add the shell's read command (Section 35.18). It reads keyboard input and waits for a RETURN. In this case, you can ignore the input; you'll use read just to make the loop wait. For example, to make the above loop prompt before each command line: 

$ for file in /usr/fran/report /usr/rob/file[23] 
> do 
> echo -n "Press RETURN to see $file--" 
> read x 
> cat -t -v $file | less 
> done 
Press RETURN to see /usr/fran/report--RETURN
    Shell runs cat -t -v /usr/fran/report | less...
Press RETURN to see /usr/rob/file2--RETURN
    Shell runs cat -t -v /usr/rob/file2 | less...
Press RETURN to see /usr/rob/file3--RETURN
    Shell runs cat -t -v /usr/rob/file3 | less...

Section 35.21 has more information about the for loop. Section 36.12 shows how to make a for loop that varies several parameters at once.

— JP

28.10 Repeating a Command with Copy-and-Paste

If you're using an xterm window (Section 24.20) or another type of terminal emulator with easy copy-and-paste functionality, that might be the easiest way to repeat all or part of a previous command line. Just select the part you want to copy, and paste it at a new prompt, adding any other text before and after pasting. This can be easier than using the shell's editing commands or history operators: what you see is what you get. Figure 28-1 shows copy-and-paste.[4]

Figure 28-1. Copying and pasting a command

figs/upt3_2801.gif

You can reuse the copied text over and over, if you want; after copying it once, paste as many times and places as you need to. Also, if you've got multiple pieces of text to copy and paste, try using a scratchpad window or xclipboard (Section 5.19).

— JP

28.11 Repeating a Time-Varying Command

figs/www.gif Go to http://examples.oreilly.com/upt3 for more information on: vis

Sometimes you find yourself repeating the same command over and over again — for example, ps (Section 24.5) to monitor the progress of your background processes, or lpq (Section 45.2) to know when your printout is finished. Instead of typing the same command repeatedly, or even using shell history (Section 30.2) to repeat it, use the vis command. For example: 

% vis ps

The vis command takes over your screen and shows the output of the initial ps command. Every 15 seconds, the command is executed again and your screen is updated with the new information. If this delay is too long for you, you can get vis to use a shorter delay using the -d option: 

% vis -d 2 ps

The information will now be updated every 2 seconds. Your screen is cleared and you are shown the output of ps. On the top line, vis tells you the command being run, how long your delay is (if not the default), and how many times it has been executed. The Exec: line is incremented every time the command is repeated. 

Command:  ps                      Delay:  2            Exec:  1

  PID TT STAT  TIME COMMAND
 2971 p1 S     0:06 -sh (csh)
 6139 p1 S     0:00 vis -d 2 ps
 6145 p1 R     0:00 ps
 3401 q0 IW    0:13 -sh (csh)
 5954 q0 S     0:01 vi ch01
14019 q5 IW    0:02 -sh (csh)
29380 r7 IW    0:00 -bin/csh (csh)
29401 rd IW    0:00 -bin/csh (csh)

vis provides a few other command-line options. The -s option is particularly neat: using -s, any lines that have changed since the last iteration are printed in standout mode.

Note that variations of this command have floated around in the public domain under several different names, such as display, rep, and watch. We found vis to be the most useful.

— LM

28.12 Multiline Commands, Secondary Prompts

All shells support multiline commands. In Bourne-type shells, a newline following an open quote (' or "), pipe symbol (|), or backslash (\) will not cause the command to be executed. Instead, you'll get a secondary prompt (from the PS2 shell variable, set to > by default), and you can continue the command on the next line. For example, to send a quick write (Section 1.21) message without making the other user wait for you to type the message, try this: 

$ echo "We're leaving in 10 minutes. See you downstairs." |
> write joanne

In the C shells, you can continue a line by typing a backslash (\) before the newline (Section 27.13). In tcsh, you'll see a secondary prompt, a question mark (?), on each continued line. The original csh doesn't prompt in this case.

Obviously, this is a convenience if you're typing a long command line. It is a minor feature and one easily overlooked; however, it makes it much easier to use a program like sed (Section 34.1) from the command line. For example, if you know you chronically make the typos "mvoe" (for "move") and "thier" (for "their"), you might be inspired to type the following command:

nroff -msSection 3.21, lpSection 45.2 

$ sed '
> s/mvoe/move/g
> s/thier/their/g' myfile | nroff -ms | lp

More importantly, the ability to issue multiline commands lets you use the shell's programming features interactively from the command line. In both the Bourne and C shells, multiline programming constructs automatically generate a secondary prompt (> in Bourne shells and ? in C shells) until the construct is completed. This is how our favorite programming constructs for non-programmers, the for and foreach loops (Section 28.9), work. While a simple loop could be saved into a shell script (Section 1.8), it is often even easier to use it interactively.

Here's an example with zsh, which makes secondary prompts that show the names of the construct(s) it's continuing. This for loop prints files from the current directory. If a filename ends with .ps, it's sent straight to the ps printer. Filenames ending with .tif are sent through netpbm (Section 45.19) filters, then to the ps printer.

case Section 35.10, echoSection 27.5

zsh% for file in *
for> do case "$file" in
for case> *.ps) lpr -Pps "$file" ;;
for case> *.tif) tifftopnm "$file" | pnmtops | lpr -Pps ;;
for case> *) echo "skipping $file" ;;
for case> esac
for> done
skipping README
  ...
zsh%

zsh's multiline editing makes it easy to go back and edit that multiline nested construct. In other shells, you might consider using a throwaway script or copying and pasting with a mouse if you have one.

—TOR and JP

28.13 Here Document Example #1: Unformatted Form Letters

figs/bomb.gif <<Section 27.16

The here document operator << (Section 27.16) is often used in shell scripts — but it's also handy at a shell prompt, especially with zsh multiline editing or a throwaway script. But you also can just type it in at a Bourne shell prompt (Section 28.12). (If you use csh or tcsh, you can either use a foreach loop (Section 28.9) or start a subshell (Section 24.4).)

The example below shows a for loop (Section 28.9) that prints three friendly form letters with the lpr (Section 45.2) command. Each letter has a different person's name and the current date at the top. Each line of the loop body starts with a TAB character, which the <<- operator removes before the printer gets the text: 

for person in "Mary Smith" "Doug Jones" "Alison Eddy"
do
   lpr <<- ENDMSG

   `date`

   Dear $person,

   This is your last notice. Buy me pizza tonight or
   else I'll type "rm -r *" when you're not looking.

   This is not a joak.

   Signed,
   The midnight skulker
   ENDMSG
done

The shell reads the standard input until it finds the terminator word, which in this case is ENDMSG. The word ENDMSG has to be on a line all by itself. (Some Bourne shells don't have the <<- operator to remove leading TAB characters. In that case, use << and don't indent the loop body.) The backquotes (Section 28.14) run the date command and output its date; $person is replaced with the person's name set at the top of the loop. The rest of the text is copied as is to the standard input of the lpr command.

— JP

28.14 Command Substitution

A pair of backquotes (``) does command substitution. This is really useful — it lets you use the standard output from one command as arguments to another command.

Here's an example. Assume you want to edit all files in the current directory that contain the word "error." Type this:

-l Section 33.6

$ vi `grep -l error *.c`
3 files to edit
"bar.c" 254 lines, 28338 characters
   ...
$

But why does this work? How did we build the incantation above? First, think about how you'd do this without using any special techniques. You'd use grep to find out which commands contain the word "error"; then you'd use vi to edit this list: 

$ grep error *.c
bar.c:  error("input too long");
bar.c:  error("input too long");
baz.c:  error("data formatted incorrectly");
foo.c:  error("can't divide by zero"):
foo.c:  error("insufficient memory"):
$ vi bar.c baz.c foo.c

Is there any way to compress these into one command? Yes, by using command substitution. First, we need to modify our grep command so that it produces only a list of filenames, rather than filenames and text. That's easy; use grep -l: 

$ grep -l error *.c
bar.c
baz.c
foo.c

The -l option lists each filename only once, even if many lines in the file match. (This makes me think that grep -l was designed with precisely this application in mind.) Now, we want to edit these files; so we put the grep command inside backquotes, and use it as the argument to vi: 

$ vi `grep -l error *.c`
3 files to edit
"bar.c" 254 lines, 28338 characters
   ...
$

You might be wondering about the difference between the "vertical" output from grep and the "horizontal" way that people usually type arguments on a command line. The shell handles this with no problems. Inside backquotes, both a newline and a space are argument separators.

The list you use with command substitution doesn't have to be filenames. Let's see how to send a mail message (Section 1.21) to all the users logged on to the system now. You want a command line like this: 

% mail joe lisa franka mondo bozo harpo ...

Getting there takes a little thinking about what Unix commands you need to run to get the output you want. (This is real "Power Tools" stuff!) To get a list of those users, you could use who (Section 2.8). The who output also lists login time and other information — but you can cut that off with a command like cut (Section 21.14): 

% who | cut -c1-8
joe
lisa
franka
lisa
joe
mondo
joe
...

Some users are logged on more than once. To get a unique list, use sort -u (Section 22.6). You're done. Just put the name-making command line between backquotes: 

% mail `who | cut -c1-8 | sort -u`

If you aren't sure how this works, replace the command you want to run with echo (Section 26.5): 

% echo `who | cut -c1-8 | sort -u`
bozo franka harpo joe lisa mondo

After using Unix for a while, you'll find that this is one of its most useful features. You'll find many situations where you use one command to generate a list of words, then put that command in backquotes and use it as an argument to something else. Sometimes you'll want to nest (Section 36.24) the backquotes — this is where the bash, ksh, bash, and zsh $( ) operators (which replace the opening and closing backquote, respectively) come in handy. There are some problems with command substitution, but you usually won't run into them.

This book has many, many examples of command substitution. Here are some of them: making unique filenames (Section 8.17), removing some files from a list (Section 14.18), setting your shell prompt (Section 4.6, Section 4.8, Section 4.14), and setting variables (Section 4.8, Section 36.23).

— JP

28.15 Handling Lots of Text with Temporary Files

Sometimes you need to execute a command with a long list of files for arguments. Here's an easy way to create that list without having to type each filename yourself — put the list in a temporary file:

'...' Section 28.14

% ls > /tmp/mikel 
% vi /tmp/mikel 
  ...edit out any files you don't want...
% process-the-files  `cat /tmp/mikel` 
% rm /tmp/mikel

I added the vi step to remind you that you can edit this list; for example, you may want to delete a few files that you don't want to process.

Possible problems: if the list is long enough, you may end up with a command line that's too long for your shell to process. If this happens, use xargs (Section 28.17). If your system doesn't have xargs, there are other workarounds doesn't that should solve the problem.

— ML

28.16 Separating Commands with Semicolons

When the shell sees a semicolon (;) on a command line, it's treated as a command separator — basically like pressing the ENTER key to execute a command. When would you want to use a semicolon instead of pressing ENTER?

Two related operators, && and || (Section 35.14), work like a semicolon, but they only execute the next command if the previous one succeeded or failed, respectively.

— JP

28.17 Dealing with Too Many Arguments

Historically, one of the more annoying things about the design of many UNIX tools was their inability to handle large numbers of arguments. For example, if you wanted to print several hundred files using lpr, you either had to pass them a few at a time, perhaps using wildcards on the command line to split the list up into shorter groups, or call lpr once per file, perhaps using find or a loop. One other method, which is still useful today, involves the use of xargs.

xargs is one of those Unix utilities that seems pretty useless when you first hear about it — but turns into one of the handiest tools you can have.

figs/www.gif Go to http://examples.oreilly.com/upt3 for more information on: xargs

If your system doesn't already have xargs, be sure to install it from the web site.

xargs reads a group of arguments from its standard input, then runs a Unix command with that group of arguments. It keeps reading arguments and running the command until it runs out of arguments. The shell's backquotes (Section 28.14) do the same kind of thing, but they give all the arguments to the command at once. This can give you a Too many arguments error.

Here are some examples:

—JP and SJC

28.18 Expect

figs/www.gif Go to http://examples.oreilly.com/upt3 for more information on: expect

Expect is a program to control interactive applications such as telnet (Section 1.21) and passwd. These and many other applications interactively prompt and expect a user to enter keystrokes in response. But you can write simple Expect scripts to automate these interactions. Then the Expect program can run the "interactive" program noninteractively. Expect can also be used to automate only parts of a dialogue, since control can be passed from the script to the keyboard and vice versa. This allows a script to do the drudgery and a user to do the fun stuff.

figs/www.gif Go to http://examples.oreilly.com/upt3 for more information on: tcl, tk

Expect programs can be written in any language but are almost always written in Tcl. Tcl is an interpreted language that is widely used in many other applications. If you already use a Tcl-based application, you won't have to learn a new language for Expect.

Tcl is a very typical-looking shell-like language. There are commands to set variables (set), control flow (if, while, foreach, etc.), and perform the usual math and string operations. Of course, Unix programs can be called, too.

Expect is integrated on top of Tcl and provides additional commands for interacting with programs. Expect is named after the specific command that waits for output from a program. The expect command is the heart of the Expect program. It describes a list of patterns to watch for. Each pattern is followed by an action; if the pattern is found, the action is executed.

For example, the following fragment is from a script that involves a login. When executed, the script waits for the strings welcome, failed, or busy, and then it evaluates [(executes) — JP] one of the corresponding actions. The action associated with busy shows how multiple commands can be evaluated. The timeout keyword is a special pattern that matches if no other patterns match in a certain amount of time. 

expect {
   "welcome" break
   "failed"  abort
   timeout   abort
   "busy"    {
        puts "I'll wait - the system is busy!"
        continue
    }
}

28.18.1 Dialback

It is surprising how little scripting is necessary to produce something useful. Below is a script that dials a phone. It is used to reverse the charges so that long-distance phone calls are charged to the computer. It is invoked with the phone number as its argument. 

spawn tip modem
expect "connected"
send "ATD$argv\r"
# modem takes a while to connect
set timeout 60
expect "CONNECT"

The first line runs the tip program so that the output of a modem can be read by expect and its input written by send. Once tip says it is connected, the modem is told to dial using the command ATD followed by the phone number. The phone number is retrieved from argv, which is a variable predefined to contain the original argument with which the script was called.

The fourth line is just a comment noting that the variable being set in the next line controls how long expect will wait before giving up. At this point, the script waits for the call to complete. No matter what happens, expect terminates. If the call succeeds, the system detects that a user is connected and prompts with login:.

Actual scripts do more error checking, of course. For example, the script could retry if the call fails. But the point here is that it does not take much code to produce useful scripts. This six-line script replaced a 60 KB executable (written in C) that did the same thing!

28.18.2 Automating /bin/passwd

Earlier I mentioned some programs that cannot be automated with the shell. It is difficult to imagine why you might even want to embed some of these programs in shell scripts. Certainly the original authors of the programs did not conceive of this need. As an example, consider passwd.

passwd is the command to change a password. The passwd program does not take the new password from the command line.[6] Instead, it interactively prompts for it — twice. Here is what it looks like when run by a system administrator. (When run by users, the interaction is slightly more complex because they are prompted for their old passwords as well.) 

# passwd libes
Changing password for libes on thunder.
New password:
Retype new password:

This is fine for a single password. But suppose you have accounts of your own on a number of unrelated computers and you would like them all to have the same password. Or suppose you are a system administrator establishing 1,000 accounts at the beginning of each semester. All of a sudden, an automated passwd makes a lot of sense. Here is an Expect script to do just that: automate passwd so that it can be called from a shell script. 

spawn passwd [lindex $argv 0]
set password [lindex $argv 1]
expect "password:"
send "$password\r"
expect "password:"
send "$password\r"
expect eof

The first line starts the passwd program with the username passed as an argument. The next line saves the password in a variable for convenience. As in shell scripts, variables do not have to be declared in advance.

In the third line, the expect command looks for the pattern password:. expect waits until the pattern is found before continuing.

After receiving the prompt, the next line sends a password to the current process. The \r indicates a carriage return. (Most of the usual C string conventions are supported.) There are two expect-send sequences because passwd asks the password to be typed twice as a spelling verification. There is no point to this in a noninteractive passwd, but the script has to do it because passwd assumes it is interacting with a human who does not type consistently.

The final command expect eof causes the script to wait for the end-of-file character in the output of passwd. Similar to timeout, eof is another keyword pattern. This final expect effectively waits for passwd to complete execution before returning control to the script.

Take a step back for a moment. Consider that this problem could be solved in a different way. You could edit the source to passwd (should you be so lucky as to have it) and modify it so that given an optional flag, it reads its arguments from the command line just the way that the Expect script does. If you lack the source and have to write passwd from scratch, of course, then you will have to worry about how to encrypt passwords, lock and write the password database, etc. In fact, even if you only modify the existing code, you may find it surprisingly complicated code to look at. The passwd program does some very tricky things. If you do get it to work, pray that nothing changes when your system is upgraded. If the vendor adds NIS, NIS+, Kerberos, shadow passwords, a different encryption function, or some other new feature, you will have to revisit the code.

Expect comes with several example scripts that demonstrate how you can do many things that are impossible with traditional shells. For example, the passmass script lets you update your password on many unrelated machines simultaneously. The rftp script provides your regular ftp client with additional commands to do recursive FTP in either direction. The cryptdir script encrypts all the files in a directory. And an amusing script is provided that lets two chess processes play each other. Expect has no limit to the number of interactive programs it can drive at the same time. The Unix system may limit Expect, though, by controlling the maximum number of processes or other system resources available.

28.18.3 Testing: A Story

Many people use Expect for testing. You can test interactive programs as easily as you can automate them. And hardware lends itself to testing with Expect, too. For example, we solved a thorny problem when we had to deal with an unreliable bank of modems. We were receiving dozens of calls each week reporting "the modem is hung." No indication of which modem, of course. And it was always too late for us to ask the user to try something to investigate the problem. The connection was gone by then. Our solution was an Expect script that connected to each modem hourly and exercised it. Any problems were recorded so that we had a clear and full history of each modem's behavior. As soon as a defective or hung modem was encountered, the Expect script would send email to the system administrator. With this script in place, reports of modem problems from our users dropped to zero.

28.18.4 Other Problems

These are just a few of the problems that can be solved with Expect. And as with all Expect solutions, recompilation of the original programs is unnecessary. You don't even need the source code! Expect handles many other problems as well. For example, Expect can wrap existing interactive tools with GUI wrappers. This means you can wrap interactive programs with graphic frontends to control applications by buttons, scrollbars, and other graphic elements. And Expect scripts work great as CGI scripts or from cron (Section 25.2) or inetd [the daemon that controls Internet services provided by a system — JP]. Finally, learning Expect may be easier than you think. Expect can watch you interact and then produce an Expect script for you. Interaction automation can't get much easier than this!

More information on Expect is available in Exploring Expect, by Don Libes, from O'Reilly & Associates.

— DL

[1]  An MH folder name can also start with an @ (at sign), but that use is less common. Besides, this is just an example!

[2]  The ~ (for Emacs) has to be quoted ((Section 27.13) when it's stored in the fignore array. Otherwise, the shell would expand it to your home directory path (Section 31.11).

[3]  If you're sending lots of mail messages with a loop, your system mailer may get overloaded. In that case, it's a good idea to put a command like sleep 5 (Section 25.9) on a separate line before the end. That will give the mailer five seconds to send each message.

[4]  This is Figure 2-3 from O'Reilly & Associates' Learning the Unix Operating System, Fourth Edition.

[5]  Actually, the shell is piping the standard output of xargs. As I said above, xargs sends the standard output of commands it runs to its own standard output.

[6]  Newer versions will accept input from STDIN, however.

CONTENTS