One of the following operating systems:

SGI server hardware:

The following Fibre Channel HBAs:

The inode monitor device (imon) is not supported on CXFS filesystems.

Do not use the IRIX fsr command; the bulkstat system call has been disabled for CXFS clients.

Are the cluster daemons running?

Is the cluster state consistent on each node? Run the clconf_info command on each server-capable administration node and compare.

Which nodes are in the CXFS kernel membership? Check the cluster status and the /var/adm/SYSLOG file.

Which nodes are in the cluster database (fs2d ) membership? See the /var/cluster/ha/log/fs2d_log files on each server-capable administration node.

Is the database consistent on all server-capable administration nodes? Determine this logging in to each server-capable administration node and examining the /var/cluster/ha/log/fs2d_log file and database checksum.

Log onto the various CXFS client nodes or use the GUI view area display with details showing to answer the following:

On the metadata server, use the clconf_info command.

Is the metadata server in the process of recovery? Look at the following file: /var/log/messages

Messages such as the following indicate that recovery status:

If filesystems are not mounting, do they appear online in XVM? You can use the following xvm command:

xvm:cluster> show vol/*

To display the hardware inventory:

irix# /sbin/hinv

If the output is not what you expected, do a probe for devices and perform a SCSI bus reset, using the following command:

irix# /usr/sbin/scsiha -pr bus_number

To configure I/O devices on an IRIX node, use the following command:

irix# /sbin/ioconfig -f /hw

To show the physical volumes, use the xvm command:

irix# /sbin/xvm show -v phys/

See the XVM Volume Manager Administrator's Guide.

To monitor system activity:

/usr/bin/sar

To monitor filesystem buffer cache activity :

/usr/sbin/bufview

---

Note: Do not use bufview interactively on a busy IRIX node; run it in batch mode.

---

To monitor operating system activity data on an IRIX node:

/usr/sbin/osview

To monitor the statistics for an XVM volume, use the xvm command:

/sbin/xvm change stat on {concatname|stripename|physname}

See the XVM Volume Manager Administrator's Guide.

To monitor system performance, use Performance Co-Pilot (PCP). See the PCP documentation and the pmie (1) and pmieconf(1) man pages.

To monitor CXFS heartbeat timeouts, use the icrash command. For example, the following command prints the CXFS kernel messaging statistics:

irix# icrash -e "load -F cxfs; mtcp_stats"
corefile = /dev/mem, namelist = /unix, outfile = stdout

Please wait............
Loading default Sial macros...........


>> load cxfs

>> mtcp_stats
STATS @ 0xc000000001beebb8
Max delays: discovery 500767 multicast 7486 hb monitor 0
hb generation histogram:(0:0)(1:0)(2:0)(3:0)(4:0)(5:0)
Improperly sized alive mesgs 0 small 0 big 0
Alive mesgs with: invalid cell 0 invalid cluster 0 wrong ipaddr 2
Alive mesgs from: unconfigured cells 100 cells that haven't discovered us 6000
mtcp_config_cell_set 0x0000000000000007
cell 0:starting sequence # 77 skipped 0
hb stats init @ 15919:(0:1)(1:478301)(2:29733)(3:0)(4:0)
cell 1:starting sequence # 0 skipped 0
hb stats init @ 360049:(0:1)(1:483337)(2:21340)(3:0)(4:0)
cell 2:starting sequence # 0 skipped 0

The following fields contain information that is helpful to analyzing CXFS heartbeat timing:

To determine IRIX kernel status, use the icrash command:

# /usr/bin/icrash
>> load -F cxfs

---

Note: Add the -v option to these commands for more verbose output.

---

To invoke internal kernel routines that provide useful debugging information, use the idbg command:

# /usr/sbin/idbg

The disk labeled as an XVM physvol was probably labeled under IRIX 6.5.6f and the system was subsequently upgraded to a newer version that uses a new version of XVM label format. This does not indicate a problem.

The cluster name had not yet been set when XVM encountered these disks with an XVM cluster physvol label on them. This is normal output when XVM performs the initial scan of the disk inventory, before node/cluster initialization has completed on this host.

The message indicates that XVM sees a disk with an XVM cluster physvol label, but that this node has not yet joined a CXFS membership; therefore, the cluster name is empty ("").

When a node or cluster initializes, XVM rescans the disk inventory, searching for XVM cluster physvol labels. At that point, the cluster name should be set for this host. An empty cluster name after node/cluster initialization indicates a problem with cluster initialization.

The first time any configuration change is made to any XVM element on this disk, the label will be updated and converted to the new label format, and these notices will go away.

For more information about XVM, see the XVM Volume Manager Administrator's Guide.

Overrun disk drives.

CXFS heartbeat was lost. In this case, you will see a message that mentions withdrawl of node.

As a last resort, do a non-maskable interrupt (NMI) of the system and contact SGI. (The NMI tells the kernel to panic the node so that an image of memory is saved and can be analyzed later.) For more information, see the owner's guide for the node.

Make the following files available: