Appendix C. Theory of Operations

This appendix is for the system administrator or operator who is familiar with the IRIX NetWorker Administrator's Guide and is interested in more information about NetWorker and how it works. You should be familiar with IRIX administrative concepts, the X Window System interface, and the nsradmin command.

The NetWorker reference pages provide complete and detailed information to help you administer NetWorker. The reference pages list the commands in alphabetical order, so that you can verify them easily.

This appendix gives a pictorial overview of the major components of NetWorker and how they are controlled. The pictures illustrate basic points; they are not meant to be detailed or precise. This appendix lists the various NetWorker reference pages that contain details about each section.

You should read this appendix while sitting at your workstation so that you can read the IRIX and NetWorker reference pages, execute NetWorker commands (especially nsradmin), and verify the pictures (using IRIX administrative commands like ps).

The basic picture elements used in this appendix are shown in FigureáC-1.

Figure C-1. Major Components of NetWorker


This appendix explains

NetWorker Media Format

The NetWorker media format is designed to meet these requirements:

  • the ability to multiplex data from several clients simultaneously to maximize performance

  • machine architecture independence (such as byte order differences) through the eXternal Data Representation (XDR) format

  • filesystem independence, allowing NetWorker to back up heterogeneous clients with different operating systems and filesystems

  • the ability to fully utilize media capacity by appending to the media until it is full, and then continuing to additional media

  • support for quick seeking on media, by writing occasional file marks

  • special handling for certain types of files via the Application Specific Module (ASM) architecture

  • the ability to track media through media labels

No other existing media format meets these requirements.

The NetWorker media format is fully documented in the mm_data(5) reference page. Third parties are welcome to adopt the format.

The NetWorker System at Rest

Four daemons continue running on the NetWorker server system even when it is at rest. The two internal server daemons, nsrmmd and nsrmmdbd, are not meant to be networker client-accessible; their only clients are forked copies of nsrd. As these internal daemons execute, they may send status messages (or “events”) to the controlling nsrd.

The controlling nsrd is the root of all NetWorker functionality. It receives status information from the internal daemons and reports the status to commands such as nsrwatch and networker. The nsrd daemon allows the browsing and editing of NetWorker resource descriptors by clients using the nsradmin command. Finally, nsrd accesses save and recover sessions from network clients and forks “agent” copies of itself to handle each session.

From a monitoring point of view, the NetWorker main window and nsrwatch are functionally equivalent. They monitor the dynamic state of the NetWorker server. A very important field in these monitors is the Pending display, which shows you what the server needs to make progress.

The networker daemon provides an X Window System graphical interface for all the system configuration tasks. The nsradmin daemon provides a character-based administrative interface to the NetWorker server that can be used from any ASCII terminal.

The four daemons that are always running on the NetWorker server and three network-based monitors are shown in FigureáC-2.

Figure C-2. NetWorker Monitors and Daemons


TableáC-1 lists the relevant reference pages.

Table C-1. Reference Pages for NetWorker at Rest

Reference Page

Comments

nsradmin(1M)

Controls all aspects of NetWorker server administration

nsrwatch(1M)

Character-based monitor

nwadmin(1M)

X Window System interface for NetWorker system administration

nsrd(1M)

NetWorker server

nsrmmd(1M)

Internal daemon interfaces to all backup devices

nsrmmdbd(1M)

Internal daemon that manages the media index


Browsing and Modifying Resources

You can configure NetWorker by using the X Window System interface or by editing the configuration information (called resources) from an ASCII terminal.

If you are not using the X Window System, you can browse and edit the NetWorker server's resources with the nsradmin command. You can either use your IRIX editor to browse and modify the NetWorker server's resource descriptors, or you can use the visual mode of nsradmin. See the nsradmin(1M) reference page for more information.

If you are using your editor, nsradmin copies the resource descriptors from the server into a temporary file on your system. When you are finished making changes, nsradmin compares any edits you may have made with the original copies and prompts you for confirmation for any additions, deletions, or changes you may have made to the resource descriptors.

Changes are passed back to the server and are immediately applied to the NetWorker server subsystems that are affected. This way, administrators do not have to kill and restart daemons or reboot systems in order to have changes applied. In addition, the changes can be applied from a client workstation using the administrator's favorite editor. All administrative aspects of a NetWorker server are handled by resource modifications.

The resource types that describe the NetWorker server configuration are shown in FigureáC-3.

Figure C-3. NetWorker Resource Types


TableáC-2 lists the relevant reference pages.

Table C-2. Reference Pages for Browsing and Modifying Resources

Reference Pages

Comments

nwadmin(1M)

X Window System interface

nsradmin(1M)

Invokes your editor on resource descriptor copies

nsr_resource(5)

Describes resource file format and types

nsrd(1M)

NetWorker server checks and applies changes


Backing Up Files

This section describes the operation of backups, called “saves.”

The save command traverses a client's filesystems and backs up a client's files, subject to directives. See “Using Directives” in Chapterá6 for information.

The save daemon first initiates a session with the server's controlling nsrd. The server accepts a connection if it originates from a privileged port on a client listed in the server's “NSR client” resource descriptor. Once the network connection is accepted, a forked agent nsrd handles all subsequent requests from the client.

After session establishment with the server, save reads files and sends a save set to its agent nsrd. Upon completion, the client calls upon the server to commit the save set. To commit the save set, the agent nsrd instructs the nsrmmd to commit the save set data to the backup media. At this point, the agent returns from the commit call. save then terminates the session and both save and the agent nsrd exit.

Under normal circumstances, in order to acquire a privileged port, save must be a “set uid root” command. Upon acquiring the privileged port, save reverts back to the invoking the user ID, unless the user is operator or the user is in the group operator.

The controlling nsrd does the original access control, then forks an agent to handle the actual saved data. By forking one agent per save session, the server can handle an arbitrary number of sessions simultaneously.

FigureáC-4 diagrams a typical save session between client and server.

Figure C-4. Typical save Session


TableáC-3 lists the relevant reference pages.

Table C-3. Reference Pages for Backing Up Files

Reference Pages

Comments

nsr(5)

Documents save directives

nsr_client(5)

Documents the resource descriptor type “NSR client”

save(1M)

Command that saves files to the server

nsrd(1M)

NetWorker server daemon checks access control

nsrexecd(1M)

NetWorker client execution daemon

rcmd(3N)

Discusses privileged ports

nsr(1M)

Security section discusses policies and issues

nsr_data(5)

Describes basic data types passed from client to server


Recovering Files

The recover command is the counterpart of save and subject to similar security and session establishment policies. Once a session is established, two major functions of recover are used: file browsing and recovery.

Saved files are browsed by using commands that are familiar to the IRIX user. Since file attributes are kept online, the browsing is possible even when no backup volume is mounted on a device. During browsing, media information may be needed if the user requests information associated with the “Versions” option in the Recover window's View menu.

As users browse, they may build a recover list. Eventually the user may give the recover command and the client submits the recovery list to the agent nsrd on the server. At this point, the file and media indexes are used to determine the backup volume(s) and position(s) of the desired file(s).

The nsrmmd requests backup volumes and reads them until the entire file recovery list is processed.

Clients browse the online indexes via remote procedure calls to their agent daemons. The device and nsrmmd are involved only when files are actually requested for recovery. Thus, NetWorker supports multiple concurrent browsers.

FigureáC-5 diagrams a typical recover session between client and server.

Figure C-5. Typical recover Session


The relevant reference page is recover(1M), which documents browsing and recovery subcommands.

Backing Up the Entire System

While it is useful that the save command allows individuals to back up their directories, an unattended backup system must back up whole systems according to predefined backup levels. The savefs command accomplishes this task.

The savefs daemon determines which backup schedule to use for a client by looking for the schedule name in the “NSR client” resource. It then consults the appropriate “NSR schedule” resource to determine which backup level to use (full, incremental, or a level 1–9. Given the save level and a filesystem, savefs saves only files within the filesystem that have been modified since the most recently recorded lesser-level save. Upon completion, the current save level and the time at which savefs began are recorded in the media database.

If the client has been configured to back up all its filesystems, savefs uses all local filesystems.

After compiling information about the local filesystems and when they were last saved, and determining the desired save level, the savefs command invokes save against each of the client's filesystems.

FigureáC-6 diagrams the savefs command saving two filesystems in parallel.

Figure C-6. Saving Two Filesystems in Parallel


TableáC-4 lists the relevant reference pages.

Table C-4. Reference Pages for Backing Up the Whole System

Reference Page

Comments

savefs(1M)

Saves the client's filesystems

nsr_client(5)

Determines the client configuration and which filesystems are backed up

nsr_directive(5)

Determines which save directives apply to the client

nsr_schedule(5)

Determines save level

save(1M)

Does the actual save, given many arguments


Automatic Network-Wide Backups

NetWorker uses the “NSR group” resource to find the start time of a network-wide backup. The pre-configured time is 3:33 each morning. Each “NSR client” resource describes the groups to which it belongs.

The controlling nsrd starts the savegroup command at the appropriate time for each group that is enabled. The savegroup daemon uses the “parallelism” attribute of the “NSR” resource to determine how many client sessions to save concurrently.

Upon completing all client saves, savegroup invokes the saveindex command to insure that the NetWorker server index is safely backed up.

The NetWorker server nsrd starts the nightly saves by invoking savegroup. Each client's savefs is initiated by savegroup in an orderly manner. After the clients are backed up, saveindex backs up the server's index.

The daemon savegroup also uses nsrexeced to run savefs on clients. This daemon uses rshd if nsrexecd is not present and running on a client.

The NetWorker server nsrd starts the nightly save by invoking savegroup, as illustrated in FigureáC-7.

Figure C-7. Starting the Nightly Save



Tip: Make liberal use of incremental save levels, which are very efficient, since they take minimal backup media space and run quickly.



Note: The daemon saveindex does a level 9 backup of the indexes to promote faster recovery after a disaster.



Tip: For unattended backups, a NetWorker server with two backup devices is worth more than twice as much as a NetWorker server with only one backup device. Often the NetWorker server with two backup devices is more productive than two NetWorker servers with only one device each. Using a jukebox is the most efficient way to complete unattended backups.

A nightly network backup is shown in FigureáC-8. (Not shown are the server's controlling nsrd and nsrmmdbd, and the client`s nsrexecd.)

Figure C-8. Nightly Network Backup


TableáC-5 lists the relevant reference pages.

Table C-5. Reference Pages for Automatic Nightly Backups

Reference Page

Comments

savegroup(1M)

Conducts the nightly network saves

nsr_service(5)

Parallelism attribute controls how many clients save at once

nsr_group(5)

Selects the time of nightly saves

nsrexecd(1M)

Remote execution system used to start savefs

rshd(1M)

Remote execution system used to start savefs if nsrexecd is not present

savefs(1M)

Saves each client's filesystems

saveindex(1M)

Saves the server's index; detailed in a later section


Event Processing

Besides reporting routine status to the controlling nsrd, the NetWorker subsystems are set to report major events. The administrator can cause any IRIX command to be invoked when an event occurs. He or she does this simply by modifying an “NSR notification” resource on the NetWorker server.

When an event occurs, the controlling nsrd inspects all “NSR notification” resources. An action is taken if the event type matches any event type listed in the hidden “event” attribute list, and if the event's priority matches any priority listed in the hidden “priority” attribute. Therefore, one event may trigger any number of actions, and one “NSR notification” may match any number of events.

All NetWorker daemons post events to the controlling nsrd. As each event is posted, the controlling nsrd matches the event against all NSR notification resources and acts on every match by executing the corresponding IRIX command. NetWorker defaults provide interfaces to the syslog system and electronic mail.

FigureáC-9 diagrams the notification subsystem.

Figure C-9. Notification Subsystem


The relevant reference page is nsr_notification(5), which describes the notification process.

Media Management

There is a working relationship between nsrmmd, the other daemons, the device resources, and the media manipulation commands. As shown, the nsrmmd daemon may be writing to one device while the operator brings a second device online. When nsrmmd needs a new backup volume, it polls the controlling nsrd for the backup volume and the device on which it is located. The nsrmmd daemon also records save set information to the nsrmmdbd daemon each time a backup volume filemark is written.

During recovery, nsrmmd queries nsrmmdbd to discover the backup volume and volume-file location of the desired data. The administrator uses the mminfo command to display information about the backup volume library.

FigureáC-10 diagrams nsrmmd and its relation to other processes.

Figure C-10. nsrmmd and Other Processes


TableáC-6 lists the relevant reference pages.

Table C-6. Reference Pages for Media Management

Reference Page

Comments

nsrmm(1M)

Media manager mounts, unmounts, deletes backup volumes

nsr_device(5)

Describes and names backup devices

nsrmmd(1M)

Writes and reads data to and from the backup devices

nsrmmdbd(1M)

Manages the backup media library

mminfo(1M)

Displays information about the backup media library


Index Management

The online index is built during saves, and is queried during browsing and recovering. The nsrls command allows the administrator to gather information about the index sizes and record counts. Index entries are purged by the nsrmm command after it deletes a backup volume from the media index. The nsrck command is automatically invoked after failure (for example, an operating system crash) to guarantee the index database consistency before the NetWorker service is enabled.

FigureáC-11 diagrams the file index and its relationship to processes.

Figure C-11. The File Index and Processes


TableáC-7 lists the relevant reference pages.

Table C-7. Reference Pages for Index Management

Reference Page

Comments

nsrmm(1M)

Media manager deletes backup volumes

nsrindexd(1M)

Manages the online index

nsrls(1M)

Displays index usage statistics

nsrck(1M)

Rebuilds indexes after a hard crash

nsr_layout(5)

Explains where all the NetWorker server's files are located


Disaster Recovery

The last act that savegroup performs each morning is backing up the NetWorker server's own index so that the index can be recovered without using itself. Besides saving the index, saveindex sends bootstrap recovery information to the printer. Should the server's index ever need to be recovered, this bootstrap information is sufficient to find the saved index on a NetWorker backup volume.

The saveindex daemon is invoked by savegroup; its job is to save index information vital to the server itself and to print recovery parameters to the line printer. Should the server's indexes ever need to be recovered, these parameters will be fed to the recoverindex command. Once the server's indexes are recovered, all other files may be recovered in the normal manner.

FigureáC-12 shows the progression of index information from saving to recovering.

Figure C-12. Index Information From Saving to Recovering


TableáC-8 lists the relevant reference pages.

Table C-8. Reference Pages for Disaster Recovery

Reference Page

Comments

savegroup(1M)

Typically invokes saveindex

saveindex(1M)

Also covers recoverindex

scanner(1M)

Used by recoverindex; reads raw NetWorker backup volumes

mm_data(5)

Describes backup media format

nsr_crash(1M)

More general information on recovering after a disk crash


NetWorker Security

When a client attempts to save files or filesystems or recover them, NetWorker starts the save, savefs, or recover program, respectively. However, before the server accepts the command to start any of these programs, the client must establish a connection with the server. Before the server accepts the connection, it must verify that the user who initiated the save or recover request has the necessary permissions:

  • The request for the connection must be made from a secure port on the user's machine. A secure port can be opened only by root, so the save, savefs, and recover programs run setuid to root.

  • The server verifies that the user executing the programs at the workstation has permission to save or recover a client's files.á For a user to have permission, one of the following criteria must be met:

    • the user's machine name must be equivalent to the client name

    • the user attempting to establish a connection must be a member of the Remote access list in the Clients window

This access control is similar to that used by the rsh (remote shell) command, except that instead of using the /.rhosts file, NetWorker uses the Remote access list in the Clients window.

Once a connection has been established, the client commands save and recover set their effective UID to the UID of the user who initiated the command so that all local filesystem and system call access is done as that user. Thus users cannot recover or back up files to which they do not have access. The exception to this rule is that the user name operator and users in the group operator have filesystem access privileges of root. Therefore, the administrator can set up a login or group for the operators who initiate backups and recovers on behalf of other users, without giving the operators root access to client systems.


Note: Since any user named operator or any user belonging to a group named operator can access all data on all clients, care should be taken in assigning these names.

NetWorker's preconfigured selections allow the clients to browse and recover only their own files. To give other clients recover access to a client's files, the administrator must explicitly add the access to the Remote access list in the Clients window, as explained earlier in this chapter.

You can further tighten access control for the client commands by turning off the set-uid bit. Doing this restricts client system use of the save, savefs, and recover commands to root. To allow access by root and operator, but not by other users, change the group ownership of these commands to operator, and set the mode bits to allow execution by owner and group.

The savegroup command initiates the savefs command on each client system in a backup group by sending a remote command request to the nsrexecd command.

The nsrexecd command runs on NetWorker client systems. This command provides a secure and restrictive way for NetWorker to start automatic backups on clients. The nsrexecd command allows you to restrict access to a select set of NetWorker servers. When you install a client, nsrexecd is started, and statements are added to the appropriate boot files to restart nsrexecd each time the client reboots.