Appendix A. OpenVault Troubleshooting

This appendix explains how to resolve error conditions, and includes a description of the possible error messages.

Most OpenVault errors are found in the log file /var/opt/openvault/server/logs/OVLOG.yyyymmdd. They can also be listed with the ov_msg  -l option; see the ov_msg(1M) man page.

Error Conditions

Errors such as tape read and write errors are associated with an application error, whereas conditions such as cannot find tape or cannot associate with driver are OpenVault errors.

Accessing OpenVault Messages

OpenVault system messages are appended to the logfile, /var/opt/openvault/server/logs/OVLOG.yyyymmdd .

To view all the warning, error, and operator intervention messages that are contained in the message log on the default server, use the ov_msg command; see the ov_msg(1M) man page.

Error Messages and Actions

Refer to the OpenVault release notes for a listing of the error messages that apply to the OpenVault system.

OpenVault Processes and Files

Table A-1 shows some important OpenVault files located in /var/opt/openvault.

Table A-1. OpenVault Configuration Files

OpenVault File

What it Controls

server/config/core_keys

File listing clients allowed OpenVault access, possibly with passwords.

clients/admin/keys

File storing security passwords for OpenVault applications.

server/logs/OVLOG.yyyymmdd

Repository of all events and errors that occur within OpenVault.

server/config/config

Configuration file for OpenVault server, MLM (media library manager).

clients/dcp/tapeN/instX/config

Configuration file for the drive named tapeN and instance name instX.

clients/lcp/libN/instY/config

Configuration file for the library named libN and instance name instY.

clients/dcp/tapeN/instX/logfile

Event and error log for the drive named tapeN and instance name instX.

clients/lcp/libN/instY/logfile

Event and error log for the library named libN and instance name instY.

Table A-2 shows some important OpenVault processes.

Table A-2. OpenVault Processes

OpenVault Process

What It Does

ovroot

If chkconfig is on, this process mediates and delegates service requests.

MLM_catalog

Grants services as requested, and manages the OpenVault catalog.

MLM_capi

Started by the server to manage each registered OpenVault application.

MLM_aapi

Started by the server to manage each OpenVault administrative application.

MLM_dcp

Started by the server to manage each attached drive and its related DCP.

MLM_lcp

Started by the server to manage each attached library and its related LCP.

DCP tapeN

The DCP (drive control program) that manages the drive named tapeN .

LCP libN

The LCP (library control program) that manages the library named libN.

ssi | lmcpd

Product-specific processes to manage ACSLS and IBM-3494, respectively.


Troubleshooting OpenVault Commands

When trying to troubleshoot OpenVault administrative commands, it's helpful to look inside the command to see what it is doing. The IRIX par command shows what system calls a program makes. This is useful, but OpenVault offers a better and more detailed way to see what an OpenVault command is doing. Before any troubleshooting can begin, debugging must be turned on in the OpenVault server, using the following command:

# /usr/sbin/ov_msg -s -t core -m debug 

Once this is done, the OpenVault server starts writing a lot more messages to the OVLOG.yyyymmdd file; so you can run a command to see what it is doing. OVLOG messages contain a lot of useful information. Here is a sample log entry:

Apr 26 17:32:44 mlm_aapi:SYSTEM:onlyInstance: (debug) [177600] WRITTEN:
response whichtask[`5'] accepted ;

This entry shows the following information:

Apr 26 17:32:44 

Current date and time.

mlm_aapi 

The OpenVault process.

SYSTEM 

The application.

onlyInstance 

There are no other such applications.

(debug) 

Level of the message; can be info, error, debug.

[177600] 

Process ID of the OpenVault process.

WRITTEN: ... 

The commands being passed along.

When an OpenVault command is run (or when any connection is made to OpenVault), a process is created to handle the connection. This is listed in OVLOG (and in the process table) as mlm_aapi . Following this process, you can see what the MLM server sees when it talks to the calling command or application as shown in Example A-1.

Example A-1. mlm_aapi Calls and Their PIDs

To get a list of individual mlm_aapi calls and their PIDs, run the following command:

# grep 'welcome version' OVLOG* | grep mlm_aapi 
Apr 26 17:32:43 mlm_aapi:SYSTEM:onlyInstance: (debug) [177600] WRITTEN:
      welcome version[`1.0'];
Apr 26 17:32:57 mlm_aapi:SYSTEM:onlyInstance: (debug) [177685] WRITTEN:
      welcome version[`1.0'];
Apr 26 17:33:27 mlm_aapi:SYSTEM:onlyInstance: (debug) [177772] WRITTEN:
      welcome version[`1.0'];

This listing shows the AAPI/CAPI reply welcome that the mlm_aapi sends to the calling command. There is only one welcome command per connection, so this should give a unique list of mlm_aapi processes.

Once the PID is found, grep can be used to locate all of the OVLOG messages that relate to that one command or connection. Example A-2, shows an ov_mount command:

Example A-2. ov_mount Search

# ov_mount -V A05444 -A dmf 

To find the specific PID for that ov_mount command, run this grep command, which looks for any OpenVault commands showing the volume A05444 and using the application dmf.

# grep A05444 OVLOG* | grep dmf 
Apr 26 14:52:24 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] READ:
    mount match [and( strEQ(VOLUME.'VolumeName' `A05444') strEQ(VOLUME.
    `ApplicationName' `dmf'))] report[VOLUME.'VolumeName' MOUNTLOGICAL.
    `MountLogicalHandle'] reportmode[value]  task[`0'];

Make sure that the time listed in the results corresponds with the time that the ov_mount command was run. Now that you found the PID for this particular command, do a grep for all messages relating to that command:

# grep 172088 OVLOG*
Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] WRITTEN:
    welcome version[`1.0'];
Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] /usr/OpenVault
    /mlm/MLM_aapi: submain(): process_peer() returned 1
Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] READ: mount
    match[and( strEQ(VOLUME.'VolumeName' `A05443') strEQ(VOLUME.
    `ApplicationName' `dmf'))] report[VOLUME.'VolumeName' MOUNTLOGICAL.
    `MountLogicalHandle'] reportmode[value]  task[`0'];
Apr 26 14:52:23 mlm_aapi:SYSTEM:onlyInstance: (debug) [172088] WRITTEN: 
   response whichtask[`0'] accepted ;
...

This shows the entire AAPI conversation from the view of the mlm_aapi or OpenVault server. All commands sent and received are shown exactly as transmitted.