CONTENTS

Chapter 39. Creating and Reading Archives

•  39.1 Packing Up and Moving
•  39.2 Using tar to Create and Unpack Archives
•  39.3 GNU tar Sampler
•  39.4 Managing and Sharing Files with RCS and CVS
•  39.5 RCS Basics
•  39.6 List RCS Revision Numbers with rcsrevs
•  39.7 CVS Basics
•  39.8 More CVS

39.1 Packing Up and Moving

The worst part of living in a nice big house is the headache of moving. The more stuff you've got room for, the more trouble it is to pack it up and take it with you.

The Unix operating system is a little bit like that. One of its real advantages is a filesystem that lets you organize your personal files into a hierarchical directory tree just like the much bigger tree that encompasses the entire filesystem. You can squirrel away all kinds of useful information into neat pigeonholes.

While your personal directory hierarchy is usually only two or three levels deep, for all practical purposes it can have as many levels as you like. And, as is true of any powerful and flexible tool, problems lie in wait for the sorcerer's apprentice. Directories and files grow increasingly complex the longer you use the system, with more forgotten files and more detailed organization.

This chapter will tackle the problems that can arise when you want to move a block of files (in one or many directories) from one place to another.

Maybe you're writing the files to a tape for safety (Section 38.3). In many cases though, this is a "backup and restore" problem. For example, if you were moving your account to another system, you might just ask the system administrator (if there is one) to archive your files to tape or floppy and restore them in the new location. Many new users are less aware that you can use the backup program tar (Section 38.2) to create online archives that you can move from one place to another.

This situation is most likely to arise in a networked environment. You might be packaging files to ship as a package to another user. The files might be going to Usenet or an archive site on the Internet, for distribution to many users. Whether you're distributing an archive to lots of people or using it for yourself, though, most of the topics we cover in this chapter will apply.

— TOR

39.2 Using tar to Create and Unpack Archives

tar (Section 38.2) is a general-purpose archiving utility capable of packing many files into a single archive file, retaining information such as file permissions and ownership. The name tar stands for tape archive, because the tool was originally used to archive files as backups on tape. However, use of tar is not at all restricted to making tape backups, as we'll see.

The format of the tar command is:

tar functionoptions files...

where function is a single letter indicating the operation to perform, options is a list of (single-letter) options to that function, and files is the list of files to pack or unpack in an archive. (Note that function is not separated from options by any space.)

function can be one of:

c

Create a new archive.

x

Extract files from an archive.

t

List the contents of an archive.

r

Append files to the end of an archive.

u

Update files that are newer than those in the archive.

d

Compare files in the archive to those in the filesystem.

The most commonly used functions are c reate, extract, and table-of-contents.

The most common options are:

v

Prints verbose information when packing or unpacking archives. This makes tar show the files it is archiving or restoring. It is good practice to use this option so that you can see what actually happens, though if you're using tar in a shell script you might skip it so as to avoid spamming the user of your script.

k

Keeps any existing files when extracting — that is, prevents overwriting any existing files contained within the tar file.

f filename

Specifies that the tar file to be read or written is filename.

z

Specifies that the data to be written to the tar file should be compressed or that the data in the tar file is compressed with gzip. (Not available on all tars.)

There are other options, which we cover in Section 38.5. Section 38.12 has more information about the order of tar options, and Section 39.3 has a lot more about GNU tar.

Although the tar syntax might appear complex at first, in practice it's quite simple. For example, say we have a directory named mt, containing these files:

rutabaga% ls -l mt
total 37
-rw-r--r--   1 root     root           24 Sep 21  1993 Makefile
-rw-r--r--   1 root     root          847 Sep 21  1993 README
-rwxr-xr-x   1 root     root         9220 Nov 16 19:03 mt
-rw-r--r--   1 root     root         2775 Aug  7  1993 mt.1
-rw-r--r--   1 root     root         6421 Aug  7  1993 mt.c
-rw-r--r--   1 root     root         3948 Nov 16 19:02 mt.o
-rw-r--r--   1 root     root        11204 Sep  5  1993 st_info.txt

We wish to pack the contents of this directory into a single tar archive. To do this, we use the following command:

tar cf mt.tar mt

The first argument to tar is the function (here, c, for create) followed by any options. Here, we use the one option f mt.tar, to specify that the resulting tar archive be named mt.tar. The last argument is the name of the file or files to archive; in this case, we give the name of a directory, so tar packs all files in that directory into the archive.

Note that the first argument to tar must be a function letter followed by a list of options. Because of this, there's no reason to use a hyphen (-) to precede the options as many Unix commands require. tar allows you to use a hyphen, as in:

tar -cf mt.tar mt

but it's really not necessary. In some versions of tar, the first letter must be the function, as in c, t, or x. In other versions, the order of letters does not matter as long as there is one and only one function given.

The function letters as described here follow the so-called "old option style." There is also a newer "short option style," in which you precede the function options with a hyphen. On some versions of tar, a "long option style" is available, in which you use long option names with two hyphens. See the manpage or info page (Section 2.9) for tar for more details if you are interested.

It is often a good idea to use the v option with tar to list each file as it is archived. For example:

rutabaga% tar cvf mt.tar mt
mt/
mt/st_info.txt
mt/README
mt/mt.1
mt/Makefile
mt/mt.c
mt/mt.o
mt/mt

On some tars, if you use v multiple times, additional information will be printed, as in:

rutabaga% tar cvvf mt.tar mt
drwxr-xr-x root/root         0 Nov 16 19:03 1994 mt/
-rw-r--r-- root/root     11204 Sep  5 13:10 1993 mt/st_info.txt
-rw-r--r-- root/root       847 Sep 21 16:37 1993 mt/README
-rw-r--r-- root/root      2775 Aug  7 09:50 1993 mt/mt.1
-rw-r--r-- root/root        24 Sep 21 16:03 1993 mt/Makefile
-rw-r--r-- root/root      6421 Aug  7 09:50 1993 mt/mt.c
-rw-r--r-- root/root      3948 Nov 16 19:02 1994 mt/mt.o
-rwxr-xr-x root/root      9220 Nov 16 19:03 1994 mt/mt

This is especially useful as it lets you verify that tar is doing the right thing.

In some versions of tar, f must be the last letter in the list of options. This is because tar expects the f option to be followed by a filename — the name of the tar file to read from or write to. If you don't specify f filename at all, tar uses a default tape device (some versions of tar use /dev/rmt0 for historical reasons regardless of the OS; some have a slightly more specific default). Section 38.5 talks about using tar in conjunction with a tape drive to make backups.

Now we can give the file mt.tar to other people, and they can extract it on their own system. To do this, they would use the command:

tar xvf mt.tar

This creates the subdirectory mt and places all the original files into it, with the same permissions as found on the original system. The new files will be owned by the user running tar xvf (you) unless you are running as root, in which case the original owner is generally preserved. Some versions require the o option to set ownership. The x option stands for "extract." The v option is used again here to list each file as it is extracted. This produces:

courgette% tar xvf mt.tar
mt/
mt/st_info.txt
mt/README
mt/mt.1
mt/Makefile
mt/mt.c
mt/mt.o
mt/mt

We can see that tar saves the pathname of each file relative to the location where the tar file was originally created. That is, when we created the archive using tar cf mt.tar mt, the only input filename we specified was mt, the name of the directory containing the files. Therefore, tar stores the directory itself and all of the files below that directory in the tar file. When we extract the tar file, the directory mt is created and the files are placed into it, which is the exact inverse of what was done to create the archive.

If you were to pack up the contents of your /bin directory with the command:

tar cvf bin.tar /bin

you can cause terrible mistakes when extracting the tar file. Extracting a tar file packed as /bin could trash the contents of your /bin directory when you extract it. If you want to archive /bin, you should create the archive from the root directory, /, using the relative pathname (Section 1.16) bin (with no leading slash) — and if you really want to overwrite /bin, extract the tar file by cding to / first. Section 38.11 explains and lists workarounds.

Another way to create the tar file mt.tar would be to cd into the mt directory itself, and use a command such as:

tar cvf mt.tar *

This way the mt subdirectory would not be stored in the tar file; when extracted, the files would be placed directly in your current working directory. One fine point of tar etiquette is always to pack tar files so that they contain a subdirectory, as we did in the first example with tar cvf mt.tar mt. Therefore, when the archive is extracted, the subdirectory is also created and any files placed there. This way you can ensure that the files won't be placed directly in your current working directory; they will be tucked out of the way and prevent confusion. This also saves the person doing the extraction the trouble of having to create a separate directory (should they wish to do so) to unpack the tar file. Of course, there are plenty of situations where you wouldn't want to do this. So much for etiquette.

When creating archives, you can, of course, give tar a list of files or directories to pack into the archive. In the first example, we have given tar the single directory mt, but in the previous paragraph we used the wildcard *, which the shell expands into the list of filenames in the current directory.

Before extracting a tar file, it's usually a good idea to take a look at its table of contents to determine how it was packed. This way you can determine whether you do need to create a subdirectory yourself where you can unpack the archive. A command such as:

tar tvf tarfile

lists the table of contents for the named tarfile. Note that when using the t function, only one v is required to get the long file listing, as in this example:

courgette% tar tvf mt.tar
drwxr-xr-x root/root         0 Nov 16 19:03 1994 mt/
-rw-r--r-- root/root     11204 Sep  5 13:10 1993 mt/st_info.txt
-rw-r--r-- root/root       847 Sep 21 16:37 1993 mt/README
-rw-r--r-- root/root      2775 Aug  7 09:50 1993 mt/mt.1
-rw-r--r-- root/root        24 Sep 21 16:03 1993 mt/Makefile
-rw-r--r-- root/root      6421 Aug  7 09:50 1993 mt/mt.c
-rw-r--r-- root/root      3948 Nov 16 19:02 1994 mt/mt.o
-rwxr-xr-x root/root      9220 Nov 16 19:03 1994 mt/mt

No extraction is being done here; we're just displaying the archive's table of contents. We can see from the filenames that this file was packed with all files in the subdirectory mt, so that when we extract the tar file, the directory mt will be created, and the files placed there.

You can also extract individual files from a tar archive. To do this, use the command:

tar xvf tarfile files

where files is the list of files to extract. As we've seen, if you don't specify any files, tar extracts the entire archive.

When specifying individual files to extract, you must give the full pathname as it is stored in the tar file. For example, if we wanted to grab just the file mt.c from the previous archive mt.tar, we'd use the command:

tar xvf mt.tar mt/mt.c

This would create the subdirectory mt and place the file mt.c within it.

tar has many more options than those mentioned here. These are the features that you're likely to use most of the time, but GNU tar, in particular, has extensions that make it ideal for creating backups and the like. See the tar manpage or info page (Section 2.9) and the following chapter for more information.

MW, MKD, and LK

39.3 GNU tar Sampler

Go to http://examples.oreilly.com/upt3 for more information on: tar

GNU tar has plenty of features; some people would say "too many." I don't agree. GNU tar has features I wish I'd had for years in more "standard" versions. This article lists my favorites. For a complete list, check the info documentation for tar.

• Section 15.7 describes how to compress an archive file you've created. If you're using GNU tar, this is even easier, since tar itself can do the compression. Simply use the z option when writing or reading archives. For example, to make the gzipped tar archive progs.tar.gz from all ".c" and ".h" files:

  % tar cvzf progs.tar.gz *.c *.h

  You can also use the long option - -gzip to get gzip compression, and the long option - -bzip2 to get bzip2 compression.

• I've made the classic mistake of archiving files with their absolute pathnames (Section 38.11). GNU tar saves you from that goof. It always stores absolute pathnames as relative paths unless you add the - -absolute-names option.

• Often I want to make a tape backup of my most recent work on a big project, but not all the thousands of files in a directory tree. The clumsy way to do that is by using find -mtime to make an include-file for the standard tar -I option. GNU tar to the rescue: its - -after-date option lets me tell it what directories to look in and how recently the files should have been changed.

• When I extract an archive, I may be writing into a directory that has other files. The - -keep-old-files option tells GNU tar not to overwrite existing files.

One caution about GNU tar: it creates ANSI-format tar archives. Extracting one of these archives with the old V7 tar can cause warning messages like "tar: unexpected EOF." But, of course, GNU tar has an option to create old-format archives: - -old-archive.

—JP and TOR

39.4 Managing and Sharing Files with RCS and CVS

How many times have you wished that you could get a copy of a file the way it looked an hour ago, or yesterday, or last year? That includes times when you just deleted the file — and, especially, when the file is too new for your computer's backup system to have made any copies of it. (You do have regular backups of your system, don't you? ;-)) RCS (Revision Control System) and CVS (Concurrent Version System) let you recover a previous version of a file from an archive. Many systems come with either RCS, CVS, or both installed already; if they don't appear to be on your system either install the appropriate package or grab the most current versions from FSF's website (http://www.fsf.org).

How does the archive get there? As you work, you periodically put a "snapshot" of the file into the archive. (The archive systems save the changes — not the whole file — so this doesn't take as much disk space as it might.) The archive remembers the date and time you store each version. You can enter a log message to describe what's changed since the last time you archived the file. You can do much more, but those are the basics.

When you need a previous version of the file, you read the archive log to decide which version is best (by date and time or by the log message). Then you use one command to get back that version. You don't have to wait for the system manager to load a tape.

Of course, these tools can't protect you from a disk crash or another disaster; that's what reliable backups are for. RCS and CVS are best for protecting you from accidentally deleting or corrupting files. But they're also great for group development projects: controlling who's working on a file, who did what when, and so on. That's especially true of CVS, which was designed to handle software developers from around the world collaborating on a project over a network — as well as a group of developers in the same office. One of my favorite features is the ability to see diff (Section 11.1) listings of what's changed between versions.

Once you get started with these tools, you'll wonder how you ever did without them. Section 39.5 explains how to protect your files with RCS. See Section 39.7 for an introduction to CVS.

— JP

39.5 RCS Basics

The Revision Control System (RCS) is a straightforward, file-based source-control system. It allows you to keep track of multiple snapshots or revisions of a file, so that you can back up to any previous version. It also allows you to note particular versions, so that you can do things such as reproduce the version of a file that you gave to someone else or released as part of a software release. Of course, it's useful for more than just software development; any time you want to change a file or set of files, revision control can be useful. To place a file under revision control using RCS:

% ci  filename 

The ci (checkin) program will prompt you for a short description of the file and commit your changes. It will by default also delete the working copy; if you want to keep a read-only copy, use the -u (unlocked) option.

To then get a working copy of the file from scratch:

% co  filename 
% co -l  filename 

The co (checkout) command will get a read-only copy of the file from RCS. If you want to edit the file, use the co -l command (the option is a lowercase L and stands for lock). While you have the file checked out and locked, no one else can edit it. When you're done, return the file to RCS (check it in) using ci again. If you use the -l option to ci, it checks in your changes and checks out a new working copy, as if you did co -l again. When you check in the file, ci asks for a brief description of your changes. These can be very useful, later, to learn the history of revisions and to find a particular revision you might want to recover; the command rlog filename gives all of the stored change descriptions.

If you create a subdirectory called RCS in the directory where you keep the code or other text files you want to protect, the RCS files will be put there for you, rather than cluttering up your main directory.

It's a good idea (but not required) to add the characters $Id $ somewhere in the file you want to place under RCS. Put this in a comment field. That is, use /* $Id $ */ in a C program and # $Id $ in a shell or Perl script. RCS will substitute the revision of the file and other useful information wherever you put $Id$ any time you check the file out; this allows you to look at a file later and know what revision it was.

If you check out a file for editing and later on decide you didn't want to change it, unlock the file using:

% rcs -u  filename 
% rm  filename 

If you want a list of all files currently checked out, use:

% rlog -L -R RCS/*

(If you don't use RCS often, you may want to store those command lines in aliases or shell functions (Section 29.1) with names like Checkout, Checkedout, and so on.) That's all there is to it!

If you are not using RCS or CVS, you should. They are an easy, ongoing way to protect yourself and do not require dozens of tapes. It is much easier just to type:

% co -r1.12  filename 

than it is to try to restore that version from backup tapes after you've deleted it. With one command, version 1.12 is restored. If it's not the right one, restore the version before or after the one you just grabbed. (If you would just like to see the file rather than get a copy, you can add the -p option to send the file to standard output. Don't forget to pipe the co -p output to less or something similar, unless it is really short.)

If you are worried that you are keeping 12 versions of the file on the disk and that this will use up a lot of disk space, don't be. RCS stores the differences between versions, not 12 separate copies of the file. It recovers earlier versions of the file on request by starting from a known point and applying patches, rather than just keeping every single revision.

Suppose you delete a file by accident. If the file is just checked out with co, it will be retrieved and marked read-only, so trying to delete the file will cause rm to ask you for confirmation. If you do delete it, you can just recover it with another co command. Suppose, however, you checked out a file with co -l, because you planned to change it. If this file gets deleted accidentally, you would lose the most recent changes. This is why you should check your files back into RCS frequently — several times a day or even more. Checking in a version whenever you make significant changes to the file, or if you make changes that would be difficult to remember, is the best insurance. Making hundreds of changes to a file without checking it back into the system is just begging for trouble.

This brief overview left out a lot of features and helpful information. For example, RCS can:

• Merge two or more peoples' work into one with rcsmerge and co -j.

• Build a tree of revisions with multiple branches and sub-branches. This lets you make and store multiple independent revisions.

• Assign an arbitrary "state" to certain revisions — for example, alpha, released, stable.

• Name some or all revisions and refer to revisions by name instead of number. This is particularly good for naming files that went into a release.

• Keep a list of users who are allowed to manipulate a particular RCS file.

To find out more, see the RCS manual pages. rcsintro(1) gives a more complete overview; manpages like ci(1) have details on the many other useful features. Finally, O'Reilly & Associates' Applying RCS and SCCS is packed with tips and techniques for using revision control in group projects (where you'll need it even more). Section 13.7 and Section 39.6 explain tools for searching RCS files.

If you're doing a larger project, take a look at Section 39.7, which discusses CVS. CVS is much better at large project coordination and provides a whole suite of useful features beyond the simple source control RCS provides.

—DJPH and BB

39.6 List RCS Revision Numbers with rcsrevs

Go to http://examples.oreilly.com/upt3 for more information on: rcsrevs

The rcsrevs script tells you all the revision numbers that are stored in an RCS (Section 39.5) file. For instance:

% rcsrevs myprog
1.3
1.2
1.1
1.2.1.1

What good is that? Here are two examples.

rcsrevs accepts rlog options to control what revisions are shown. So rcsrevs -r2 somefile would list only revisions 2.0 and above, rcsrevs -sbeta would list the revisions in beta state, and so on.

— JP

39.7 CVS Basics

The Concurrent Version System, or CVS, is a version control system designed to support complex project structures or groups of people who are working together on a common set of files. Where RCS (Section 39.5) deals only with individual files, CVS allows you to work with entire projects as a whole. As we have mentioned before, while source control systems were originally developed primarily for use in developing software, they make a great deal of sense any time you want to keep track of changes to files. CVS is good for keeping track of changes to source files for a book or configuration files for qmail or apache, or for any number of other day-to-day tasks.

CVS stores its archives in a directory called a cvsroot. You tell CVS where to find the repository you want to use by setting the CVSROOT environment variable or using the -d option:

% setenv CVSROOT /home/cvsroot
% cvs checkout conf

% cvs -d /home/deb/cvs checkout book

Within a cvsroot are one or more repositories. Each repository is associated with a particular project (or in the case of a very complex project, a piece of a project). To work on a project, you much check out its repository to create a working area using cvs checkout, as in the example above. CVS is helpful and remembers which cvsroot you used for a particular checkout; future commands within that working area automatically use the right repository. For the record, the working area's cvsroot overrides the CVSROOT environment variable; the -d option overrides them both.

Once you have a working area, you have a writable copy of every file in that project. Edit to your heart's content. To incorporate changes made by other people, or see what you've changed, use cvs update:

% cd book
% cvs update
cvs update: Updating .
U ch18.sgm
M ch39.sgm

CVS update tells you a bit of information about each file that it touched or needs to touch. A U means that it updated your working copy from the repository; if you had also changed that file, it means that CVS successfully merged their changes with yours. A M means that you've modified that file in your working area.

To push your modifications into the repository, you use cvs commit. As the name suggests, this commits your changes. Generally you'll want to do this often, so that you aren't set back very far if you delete a file accidentally or make a change you later decide you don't want.

CVS does more, of course. For example, cvs log lets you read the log that shows differences between two revisions. cvs diff lets you see the differences between two revisions by comparing them with diff (Section 11.1). cvs add (followed by cvs commit) adds a new file or directory to the repository. cvs remove removes a file or directory; be sure to remove any local copy first, or use cvs remove -f to have CVS remove your local copy for you. cvs init initializes a new cvsroot, and cvs import creates a new repository. Notifications can be emailed automatically when a file is changed. Part or all of the repository can be made read-only for all but a few users — so you can share files freely but prevent unauthorized changes. O'Reilly's CVS Pocket Reference gives a summary of all this and much more about CVS.

— DJPH

39.8 More CVS

Here's a slightly more complex example of how to use CVS. I'm working on this book, via CVS, with my two main coauthors (who are on the east and west coasts of the United States). The repository, which has almost 1,000 files, is on a computer in the O'Reilly office in Massachusetts.

—JP and DJPH

CONTENTS