This chapter describes the following troubleshooting topics:
Addressing drive, job, and daemon issues
Using tracing
Resolving common problems
Occasionally, you may experience problems with the hardware or the software while running magnetic tapes. If so, there are certain steps you should take to try to clear the user, job, tape drive, or the TMF daemon itself. This section describes those steps and identifies TMF daemon files that you may encounter.
If a tape drive appears to be hung, but the TMF daemon is still responding to commands such as tmstat(1) and tmgstat(8), you can use the tmfrls (8) command to clear the user's tape reservation. If this method does not work, try the tmclr(8) command.
If the problem appears to be hardware related, free the user by the preceding method (check the result with the tmstat(1) command). Then configure the drive down with the tmconfig(8) command, and discuss the problem with the appropriate hardware personnel.
If the TMF daemon (see tmdaemon(8)) is hung (that is, no tapes are moving nor are there any responses from any tape commands), you must take the TMF daemon down. First try the tmstop(8) command. If this command does not work, determine the process identifier of the TMF daemon (by using the ps(1) command), and enter the following kill(1) command:
kill -2 pid |
The pid argument of the kill(1) command is the process identifier of tmdaemon(8). If the previous command does not work, enter the following:
kill -9 pid |
A number of files throughout the system relate to tapes. This section deals with those files specific to the TMF daemon.
All of the following TMF commands and processes reside in the /usr/tmf/bin directory:
During the course of its activity, the TMF daemon and its components write a number of trace files, which are located in the /usr/spool/tmf/trace directory. Table 4-1, describes this subset, and Example 4-1, shows how you use the tmstat(1) command to identify a tmfxxx file.
File | Description |
|---|---|
avr_device_name | Each tmavr process records events in a trace file based on the device name it is monitoring. If AVR is active for the s4781s0 device, the relevant trace entries for the tmavr process are in avr_s4781s0. |
daemon | This file contains all activity traced by the TMF daemon. It is the main TMF daemon trace file. The tmset(8) command must be issued with the -T option set to off in order to disable traceing and may impact problem diagnosis as minimal tracing may not provide enough information to resolve problem situations. A site must weigh the benefits of disabling tracing against the potential drawbacks. Disabling tracing does enable the TMF daemon to run more efficiently. |
daemon.stdout, daemon.stderr | These files contain any information that goes to standard output or error. They are in the /usr/spool/tmf directory. The daemon.stderr file is especially helpful in tracking down problems as it contains error messages as well as informational messages pertaining to various administrative commands. |
ldrname | Each media loader also has its own trace file. The name of this file corresponds to the loader name as defined in the tmf.config file. |
tmfxxx | Once a tape is assigned a drive, subsequent traces specific to that process are logged in a tmfxxx file. The final three characters of the trace file can be determined from the stm field of the tmstat(1) command. |
In this tmstat output, the traces for drive s4781s0 are in the tmf002 file. Leading zeros are added to the stream number to make it a 3-character number to create the tmfxxx file name.
armadillo%>tmstat
user sess group a stat device stm rl ivsn evsn blks NQSid
STK9490 - idle s9490s4
STK9490 - idle s9490s1
bar 3854 STK4781 - assn s4781s0 2 is 002335 002335 1
STK4781 - idle s4781s1
STK4781 - idle s4781s2
STK4781 - idle s4781s3
STK4781 - idle s4781s4
|
In addition, communication pipes are maintained within the /usr/spool/tmf directory. If the TMF daemon abnormally terminates, its core file is also saved in the directory.
The message daemon logs can provide insight into tape problems. These log files are generally saved and maintained in the /usr/spool/msg directory. All operator interaction is saved in the msglog.log file. In addition, a debug log for the message daemon is in the dbglog.log file.
Using tracing can help identify and resolve tape problems. The tmcollect(8) utility enables you to collect the trace information needed.
The tmcollect(8) utility collects TMF information. A user with root permission may run this script when a tape-related problem occurs. The information is placed in a separate directory so that it can be easily packaged and shipped for offline analysis. For the collected information to be of optimal use, TMF tracing should be enabled. For more information about this administration command, see the tmcollect(8) man page.
Before anything is copied to the information directory, the tmcollect(8) utility attempts to determine whether the TMF daemon is in its normal state, and if not, runs a few checks for known hang situations.
The tmcollect(8) utility should be executed to gather information once trouble with the TMF daemon is suspected prior to attempting to terminate the TMF daemon.
TMF tracing is turned on by default. All child processes created by the TMF daemon have tracing enabled. While tracing is a very important tool for debugging TMF problems, it uses additional CPU time. Tracing can be turned on and off by issuing the tmset(8) command. To turn tracing off, enter the following command:
tmset -T off |
To turn tracing on, enter the following command:
tmset -T on |
If the stability of TMF at a site has been established, tape tracing may be unnecessary overhead. The CPU cycles saved by turning tracing off depends on the mix of jobs submitted, because some tape operations generate more trace information than others.
When tracing is turned off, the TMF daemon and its child processes still trace entry to and exit from child processes and abnormal termination of tape processes. Abnormal terminations include those induced by the operator and terminations caused by errors within TMF. A tape mount request canceled by an operator or interrupted user job is considered an abnormal termination induced by the operator.
The option of turning TMF tracing off allows sites at which TMF is stable to reduce substantially the system and user time used by the TMF daemon. This gain in system and user time must be weighed with the knowledge that some error information and all trace information will be lost in case of a TMF daemon problem.
The only way to analyze a problem is to turn tracing on, resubmit the job, and collect traces when the problem reappears.
To obtain a complete picture of a problem, save trace information as soon as possible after you identify an error situation. You can use the tmcollect(8) utility to aid in the data gathering process.
This utility saves all the pertinent trace files in /var/spool/tmf. If the TMF daemon is not hung, the TMF command output is also saved. When you execute the utility, you are asked to comment on how the system was behaving at the time tmcollect(8) was run.
All of the trace files are circular. For instance, if a particular tape drive is hung, by the time it is noticed the TMF daemon trace has probably been overwritten. However, the device trace should provide some useful information. By default, the device traces are 409600 bytes in length while the daemon file is 10 times that value (the default is 4096000 bytes). You can configure this parameter by specifying the trace_file_size option in the OPTIONS statement in the TMF configuration file. For more information, see the tmf.config(5) man page.
Each time a TMF daemon routine is entered, tracing for that routine begins. Additional tracing may also exist which provides more information for software engineering in case problems occur. By using this information, the paths that the software took to perform various tape functions can be followed.
Information is also written into the respective TMF daemon device traces (tmfxxx). In addition, there are trace files for esinet, stknet, and ibmnet. By using all of the appropriate traces, you can obtain the entire picture of what was happening when a failure occurred.
Example 4-2 shows the information you can obtain from a trace line.
This example identifies and describes each trace line segment.
10:59:58 151257598.1241 1450 tmmsp media_select function entered ^^^^^^^^ ^^^^^^^^^^^^^^ ^^^^ ^^^^^ ^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^....... AAAAAAAA BBBBBBBBBBBBBB CCCC DDDDD EEEEEEEEEEEE FFFFFFFFFFFFFFFF....... |
The fields in this line are labeled as follows:
This section identifies some common tape problems that you may encounter and some possible solutions.
This error indicates that you issued a tmrsv(1) command for a device group that does not exist, or that you attempted to reserve more devices than are currently configured up.
This message is returned when a tmmnt(1) command has been issued, but has not yet been satisfied because a requested device type is not available. The command will be satisfied once a device is made available either by the operator configuring one up or by a currently running job releasing its resources.
This error is returned when the file specified with the -f parameter on the tmmnt(1) command (or -p if -f is not present) does not exist on a labeled tape. When a labeled tape is created, the lower 17 characters specified by the -f (or -p) parameter are written into the HDR1 label. Subsequent attempts to read that tape file must include the correct file identifier. The file identifier is not checked if the check_file_id option is set to NO in the tmf.config file.