The most important maintenance task for a system administrator is ensuring frequent, reliable backups of essential disk storage. Because of this task's preeminent importance, we devote this chapter to it. The next chapter discusses other regular maintenance tasks.
ClearCase does not include any data backup tools. All ClearCase data is stored in standard files, within standard directory trees; thus, you can use any backup tools at your disposal. The standard UNIX utilities tar(1) and cpio(1) are well-suited to backing up ClearCase data structures.
By default, a VOB storage directory is wholly contained in a single directory tree, which resides in a single disk partition. If you use a file-oriented backup tool, you need only specify the VOB storage directory to ensure a complete backup. If you use a disk-partition-oriented backup tool, you need only specify the partition name.
|Note: The commands listed in the following sections don't require a ClearCase view context. The commands “don't care” whether or not a VOB is activated (mounted) on client hosts.|
To determine the location of a VOB's storage directory, use lsvob:
% cleartool lsvob -long /vobs/flex VOB on host: ccsvr01 VOB server access path: /vobstore/flex.vbs (use this information if your backup program works locally) . . Tag: /vobs/flex Global path: /net/ccsvr01/vobstore/flex.vbs . . Region: uno (use this information if your backup program runs over the network)
Specify the appropriate pathname to your backup program. With a program that backs up entire disk partitions, you'll need to determine what partition the VOB storage directory resides in.
A backup will represent a self-consistent snapshot of a VOB storage directory's contents only if the VOB remains unmodified while the backup program is working. You need not unmount a VOB to ensure this—just lock it before backing it up, and unlock it afterward.
% cleartool lock -vob /net/ccsvr01/vobstore/flex.vbs Locked versioned object base "/net/ccsvr01/vobstore/flex.vbs". <perform backup> % cleartool unlock -vob /net/ccsvr01/vobstore/flex.vbs Unlocked versioned object base "/net/ccsvr01/vobstore/flex.vbs".
If you use a file-oriented backup program, you may wish to back up only some subdirectories within the VOB storage directory, in order to save time. Use the guidelines in Table 9-1 (carefully!) to determine the relative importance of the various components of a VOB:
Table 9-1. VOB Components for Partial Backups
Importance for Backup
top-level VOB storage directory
VOB database subdirectory
source storage pools
derived object storage pools
important, but not essential
cleartext storage pools
Backing up derived object storage pools is not crucial because, by definition, DOs can be rebuilt from sources. The importance of backing up these pools may change over time:
In the early stages of a project, when the source base in changing rapidly, the useful life of most derived objects is very short. Omitting DO storage pools from a backup regimen probably won't be much noticed by developers.
When a project is relatively stable, a VOB's DO storage pools contain many often-reused objects. At this stage, a complete build of a software system might wink-in virtually all DOs, rather than building them. Loss of a DO storage pool might increase the time requirement for such a complete system build by an order of magnitude.
|Caution: A shared derived object has two parts: an object in the VOB database and a data container in a DO storage pool. Loss of DO data containers (for example, through failure to back them up) throws the VOB's database “out of sync” with its DO storage pools. To resynchronize, you must delete all the “dataless” DOs from the VOB database, using rmdo.|
Backing up cleartext storage pools is not important at all, because they are merely caches that enhance performance. ClearCase type managers recreate cleartext data containers automatically, as necessary.
Suppose you decide to back up all but the cleartext storage pools of the VOB located at /vobstore/flex.vbs. This would entail:
Backing up the files, but not the subdirectories of /vobstore/flex.vbs.
Backing up the entire contents of directory /vobstore/flex.vbs/.identity, which contains the VOB's ownership and group membership information.
Backing up the entire directory tree at /vobstore/flex.vbs/db, which contains the VOB database.
Backing up the entire directory tree at /vobstore/flex.vbs/s, which contains the VOB's source storage pools.
Backing up the entire directory tree at /vobstore/flex.vbs/d, which contains the VOB's derived object storage pools.
If a VOB has remote storage pools, then its on-disk storage is probably not wholly contained within a single disk partition. It is quite likely that the storage is distributed among two or more hosts. In this situation, the overall procedure is:
For Step #3, use the lspool command in any view to determine which storage pools are remote, and their actual locations:
% cleartool lspool -vob /vobs/flex 13-Jan.16:58 vobadm pool "cdft" "Predefined pool used to store cleartext versions." 26-Jan.22:02 vobadm pool "cltxt01" "remote cleartext storage pool for 'flex' VOB" (only remote storage pool)
13-Jan.16:58 vobadm pool "ddft" "Predefined pool used to store derived objects." 13-Jan.16:58 vobadm pool "sdft" "Predefined pool used to store versions." % c lspool -long -vob /vobs/flex cltxt01 pool "cltxt01" . . pool storage global pathname "/vobstore/flex.vbs/c/cltxt01" . .
If you are not sure which storage pools are remote, enter a lspool –long command to list the “pool storage global pathname” of every pool, and examine these pathnames to determine which ones specify remote locations.
The following procedure restores a VOB backup without disrupting ongoing work. We assume that the VOB is the same one discussed in “Backing Up a VOB”.
|Note: You can restore a VOB to a new location, on the same host or on another host. In this case, you must reregister the VOB at its new location (Step #8).|
Unmount the VOB—The VOB must be unmounted on each client host. For example:
% cleartool umount /vobs/flex
Go to the VOB host—Log in, as the root user, to the host where the VOB storage directory resides:
% rlogin ccsvr01 -l root Password: <enter root password> #
Check disk space availability—Make sure there is enough free space in the VOB's disk partition to load the backup copy:
# cleartool space /vobstore/flex.vbs Use(Mb) %Use Directory 27.0 2% VOB database /net/ccsvr01/vobstore/flex.vbs 33.0 3% cleartext pool /net/ccsvr01/vobstore/ flex.vbs/c/cdft . . -------- ---- ------------------------------------------ 312.9 28% Subtotal 828.4 74% Filesystem /net/ccsvr01/vobstore (capacity 1115.1 Mb)
If the available space is insufficient, delete the VOB storage directory, or use other means to make enough space available.
Shut down ClearCase on this host—This ensures that ClearCase processes associated with the VOB are terminated. For example:
# /etc/rc.atria stop
|Note: The name of the ClearCase shutdown script varies from system to system. See the init_ccase manual page.|
Move the original VOB aside—If it still exists, rename the VOB storage directory:
# mv /vobstore/flex.vbs /vobstore/flex.OLD
Restart ClearCase on this host—This reenables use of any other VOBs located there. For example:
# /etc/rc.atria start
Load the backup—Restore the VOB storage directory from the backup medium. For example:
# mkdir /vobstore/flex.vbs # cd /vobstore/flex.vbs # tar -xvp
|Note: Each VOB storage area includes a directory named .identity, which stores files with special permissions: the setUID bit is set on file uid; the setGID bit is set on file gid. You must preserve these special permissions when you restore a VOB backup:|
If you used tar(1) to back up the VOB, use tar's -p option when restoring the VOB. In addition, make sure to enter the tar command as the VOB owner or as the root user.
If you used cpio(1) to back up the VOB, no special options are required in the cpio command that restores the backup data.
If the VOB has remote storage pools, restore the backups of these pools at this point.
Reregister the VOB—This is necessary only if you restored the VOB to a new location. For example, if you restored the VOB to new location /vobst_aux/flex.vbs:
# cleartool unregister -vob /vobst_aux/flex.vbs # cleartool register -vob /vobst_aux/flex.vbs (Yes, unregister followed by register) # cleartool mktag -vob -replace -tag/vobs/flex/ \ vobst_aux /flex.vbs
Mount the restored VOB on one host—If the VOB host is also a client host, activate it there. Otherwise, use some other ClearCase host:
# cleartool mount /vobs/flex
Unlock the restored VOB—The VOB should have been locked before it was backed up. (See “Ensuring a Consistent Backup”.)
# cleartool unlock -vob /vobstore/flex.vbs Unlocked versioned object base "/vobstore/flex.vbs".
Mount the restored VOB on all hosts—The restored copy of the VOB is now ready to use. Have users remount the VOB on their workstations, using cleartool mount.
|Note: This section applies in any situation where a VOB's database has been “rolled back” to a previous state.|
After you restore a VOB from backup, its VOB database may be out-of-date with respect to certain derived objects. The “old” database won't know about any DOs that were created in subsequent ClearCase builds. This will cause errors during hierarchical builds in which those “late-arriving” DOs are reused to construct higher-level targets:
======== Rebuilding "libbld.a"======== building libbld.a rm -f libbld.a rm -f /vobs/atria/sun5/pvtlib/libbld.a /opt/SUNWspro/bin/cc -c -o ... ar cq libbld.a bld.o bld_pp.o ... Will store derived object "/vobs/proj/sun5/libbld_V.o" Will store derived object "/vobs/proj/sun5/libbld.a" clearmake: Error: INTERNAL ERROR detected and logged in "/var/adm/atria/error_log".
The error_log file shows:
Sunday 12/19/93 15:55:02. host "scandium", pid 440, user "chase" Internal Error detected in "../bldr_vob.c"line 114 clearmake/cm/bldr_vob: Error: VOB "scandium:/vobs/proj" missing config record for derived object (OID) "0b5759d0.fb1811cc.a0af.08:00:69:02:2e:aa"
To reestablish the view's consistency with the VOB:
Determine which DOs are causing the inconsistency—The cleartool ls command annotates them with [no config record]:
% cleartool ls bldr_comm.ugh@@09-Dec.18:26.287028 bldr_cr.msg.o [no config record] bldr_cr.o [no config record] bldr_cr.ugh [no config record] bldr_cr_cache.msg.c@@24-May.20:51.42929 . .
Remove the offending DOs—Use the standard rm(1) command:
% rm bldr_cr.msg.o bldr_cr.o bldr_cr.ugh
Backing up views is similar to backing up VOBs, but is often simpler:
Users may create views under their home directories. In this case, whatever backup regimen you have in place to back up users' home directories will automatically pick up their view storage directories, as well.
Views do not have multiple storage pools, some of which may be local and others remote. A view has a single private storage area, its .s subdirectory, which can be either local or remote.
It never makes sense to attempt a partial backup of a view storage directory—all the data is important, because it's the only copy of one or more users' current work.
Use the following procedure to back up a view.
Determine the location of the view storage directory—Use the lsview command:
% cleartool lsview akp_vu View on host: neptune View server access path: /home/akp/views/akp.vws (use this information if your backup program works locally) . . Tag: akp_vu Global path: /net/neptune/home/akp/views/akp.vws . . Region: dvt (use this information if your backup program runs over the network)
Ensure self-consistency of the backup—A view cannot be locked with the lock command. To guarantee that a view is inactive when it is backed up:
Warn ClearCase users that the view is about to become inactive.
Use kill(1) to terminate the associated view_server process on the host where the view storage directory resides.
A user's subsequent setview or startview command will invoke a new view_server process. If you are not confident that user's will refrain from such activity, rename the view storage directory. (In the following steps, we assume that you have not renamed it.)
Determine whether the view has remote storage—You can use a standard ls(1) command:
% cd /home/akp/views/akp.vws % ls -ld .s ... .s -> /net/ccsvr04/viewstore/akp.stg (symbolic link indicates remote private storage area)
Enter the backup command(s)—If the view's private storage area is local, then a single command can back up the entire view storage directory tree. Use the local address or the network-wide address listed by lsview in Step #1:
% cd /home/akp/views % tar -cv akp.vws
If the view's private storage area is remote, you need to perform a second backup (typically, using a different backup tape):
% tar -cv /net/ccsvr04/viewstore/akp.stg
Reactivate the view—If you renamed the view storage directory in Step #2, restore it to its original name. Inform users that they may resume using the view, with setview and startview commands.
Use the following procedure to restore a backup view storage directory. We assume that the view is the same one discussed in “Backing Up a View”.
|Note: You can restore a view to a new location, on the same host or on another host. In this case, you must reregister the view at its new location (Step #5).|
Go to the view host—Log in to the host where the view storage directory resides:
% rlogin neptune
Check disk space availability—Make sure there is enough free space in the view's disk partition to load the backup copy. If necessary, delete the view storage directory, or use other means to make enough space available.
Move the original view aside—If it still exists, rename or delete the view storage directory:
# mv /home/akp/views/akp.vws /home/akp/views/akp.vws.OLD
Load the backup—Restore the view storage directory from the backup medium. For example:
% cd /home/akp/views % tar -xv
If the view's private storage area is remote, restore its backup, too.
Reregister the view—This is necessary only if you restored the view to a new location. For example, if you restored the view to new location /usr2/akp.vws:
# cleartool unregister -view /usr2/akp.vws # cleartool register -view /usr2/akp.vws (Yes, unregister followed by register) # cleartool mktag -view -replace -tag akp_vu /usr2/akp.vws
Reactivate the view—The restored copy of the view is now ready to use. Use a setview or startview command to start a view_server process. Inform the user(s) of the view that it is available again.