You should be able to use the tools discussed in “Troubleshooting Tools”.

You should have a good understanding of the log files discussed in “Status in Log Files” in Chapter 14.

When you encounter a problem, identify the cluster status by answering the following questions:

Before you start configuring another new cluster, make sure no nodes are still in a CXFS kernel membership from a previous cluster. Enter the following to check for a cmsd kernel thread:

server-admin# ps -ef | grep cmsd

If the output shows a cmsd kernel thread, perform a forced CXFS shutdown by entering the following:

server-admin# service cxfs stop

Then check for a cmsd kernel thread again.

After waiting a few moments, if the cmsd kernel thread still exists, you must reboot the machine or leave it out of the new cluster definition. It will not be able to join a new cluster in this state and it may prevent the rest of the cluster from forming a new CXFS kernel membership.

To determine if a node is fenced, log in to a server-capable administration node and use the cxfs_admin status command or the hafence(1M) command.

The following messages are logged when fencing changes:

Raising fence on cell cellID (nodename)
Lowering fence on cell cellID (nodename)

The quorum master is the server-capable administration node with the lowest node ID number (not cell ID). To confirm which node is the quorum master, look in the /var/log/cxfs/fs2d_log file for the most recent quorum message, such as the following example, which shows that node3 is the quorum master:

Tue Jul  7 12:52:41.490 cxfsopus4 fs2d  - Checking quorum with 4 members for any unknown members.
Tue Jul  7 12:52:41.490 cxfsopus4 fs2d  -  All quorum member machines known, master is node3

To locate the problem, do the following:

Brocade switch problems can cause CXFS to behave abnormally. For easier troubleshooting, use the syslogdipadd function on the switch to redirect its syslogd information to up to six potential metadata servers in the cluster. SGI recommends logging to at least two potential metadata servers on which you troubleshoot issues and look for error messages. The syslogd information is the same as that given by errshow command on the switch.

For example, on each switch, define the metadata server nodes MDS1 and MDS2 to which the switch can redirect its syslogd output:

switch:admin > syslogdipadd ipaddress_MDS1
switch:admin > syslogdipadd ipaddress_MDS2

The entries from the switch can be sorted because they are prefixed by the switch name, which is standard syslogd behavior.

Understand the following physical storage tools:

# lsscsi

Probe the LUNs with echo

For all nodes with Fibre Channel or serial-attached storage (SAS) connections and some nodes with InfiniBand connections, use the following command to probe the LUNs on the specified SCSI host hostX:

# echo "- - -" > /sys/class/scsi_host/hostX/scan

Each "-" character is a wildcard for bus, target, and LUN, respectively.

---

Note: InfiniBand drivers and infrastructure on Linux are moving targets and the behaviors encountered can be distribution-specific; their documentation beyond the scope of this guide. In some Linux distributions, there is an issue with the InfiniBand srp driver such that issuing the above command can cause redundant targets to be created per InfiniBand host. For those distributions, the following is the preferred command to probe each selected target for new LUNs # echo "bus target -" > /sys/class/scsi_host/hostX/scan To determine the appropriate bus and target values, use the lsscsi output showing host:bus:target:lun for already-discovered targets. You may need to use the srp_daemon command to dynamically add new targets.

---

linux# /sbin/xvm show -v phys/

This section discusses the following:

# /sbin/xvm

server-admin# /usr/cluster/bin/cdbreinit 

server-admin# /usr/cluster/bin/cxfs-config -all -check

Understand the cluster control tools:

Understand the following networking tools:

Understand the following cluster/node status tools:

Understand the following performance monitoring tools:

For system dump analysis, use the crash(8) tool provided with SLES or RHEL.

To enable the collection of crash dumps, do the following:

For more information, see the crash(8) man page.

---

Note: On systems with CXFS installed, a set of crash extensions is available. These extensions are loaded with the extend sgidbg command incrash. The sgi-cxfs-kmp-kernelrev and cxfs_utils RPMs are required.

---

Before reporting a problem to SGI, do the following:

If the GUI will not run, check the following:

If you create new slices on a previously sliced disk that have the same starting blocks as slices already existing on the disk, and if the old slices had filesystems, then the GUI will display those old filesystems even though they may not be valid.

If the cxfs_admin output appears to be stale (such as after you manually change the port status, in which case the CXFS database is not informed), you can update the CXFS database by running the following command:

server-admin# /usr/cluster/bin/hafence -U

If you cannot log in to a server-capable administration node, you can use one of the following commands, assuming that the node you are on is listed in the other nodes' .rhosts files:

# rsh hostname ksh -i
# rsh hostname csh -i

The following message indicates a problem with client membership:

Error -N reading mesg header channel X cell Y

where N can be one of:

The following series of messages indicate that a client has lost membership (line breaks added here for readability):

Mar 15 10:55:35 5A:mvcxfs2 kernel: Error -61 reading mesg header channel 0 cell 4 (mvcxfs17) 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:35 4A:mvcxfs2 kernel: Error receiving messages from cell 4 (mvcxfs17) tcpchannel 0 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:36 5A:mvcxfs2 kernel: Error -61 reading mesg header channel 1 cell 4 (mvcxfs17) 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:36 4A:mvcxfs2 kernel: Error receiving messages from cell 4 (mvcxfs17) tcpchannel 1 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:36 5A:mvcxfs2 kernel: Error -61 reading mesg header channel 1 cell 4 (mvcxfs17) 
  [priority 2 at 163.154.17.173 via 163.154.17.48]
Mar 15 10:55:36 4A:mvcxfs2 kernel: Error receiving messages from cell 4 (mvcxfs17) tcpchannel 1 
  [priority 2 at 163.154.17.173 via 163.154.17.48]
Mar 15 10:55:36 4A:mvcxfs2 kernel: Transport failure cell 4 
  [priority 2 at 163.154.17.173 via 163.154.17.48] 0 of 2 interfaces up
Mar 15 10:55:36 6A:mvcxfs2 kernel: Heartbeat Monitor:Failure time-stamp 295789 ticks:Last heartbeat 
  time-stamp 289940 ticks:Time-stamp delta 5849 ticks (5 seconds):Heartbeat timeout 5120 ticks (5 seconds)

The Error receiving and Error reading messages indicate that the message channel went down. The last message, which includes the Heartbeat Monitor string, contains other strings that give clues as to why the channel was disconnected. Table 15-1 lists all of the possible strings that may be included.

In the above example, the last message indicates that there is a heartbeat timeout because the string Heartbeat Monitor is included. The message also indicates that the error was detected at 295789 ticks (Failure time-stamp string) and that the configured timeout is 5120 ticks or 5 seconds (the Heartbeat timeout string). The delta is 5849 ticks or 5 seconds (the Time-stamp delta string), therefore it is a heartbeat timeout because the delta is greater than the configured heartbeat timeout.

If you are unable to raise the fence on a node, it may be that the switch ports are unable to determine the WWPN. See “Hardware Changes and I/O Fencing” in Chapter 12.

If you are unable to define a node, it may be that there are hostname resolution problems. See “Hostname Resolution and Network Configuration Rules” in Chapter 6.

If a node is detected in the system log file but it never receives a Membership delivered message, it is likely that there is a network problem. See Chapter 8, “Postinstallation Steps”.

If you experience problems involving a node being inappropriately taken out of the cluster membership (reset, fenced, or shut down, depending upon how the node fail policy is set), it may be that the CXFS kernel heartbeat timeout setting is inappropriate, especially if your cluster includes a large SGI ia64 system (one with more than 64 processors). If you suspect this problem, you should examine the SYSLOG messages to determine the current kernel heartbeat timeout. Membership issues can involve either:

Every kernel heartbeat multicast packet includes a counter value that monotonically increases with every heartbeat sent by the system. You can use a snoop/tcpdump of the private network to look at the counter and determine if the issue is a missing heartbeat or a late heartbeat. A nonconsecutive heartbeat count indicates that the heartbeat was issued by the client-only node but never received by the server-capable administration node. It generally indicates a problem somewhere in the network stack or physical network layer.

Most client-only node heartbeat timeouts are caused by late heartbeats. This is a heartbeat multicast that is not sent out by the client-only node in a timely fashion. It is a strong indicator that the client-only node is suffering from some sort of resource constraint issue. Many common resources are subject to this problem, and detailed system performance analysis is usually required to determine the exact nature of the failure. The following are common problem areas:

To avoid these problems, see “Avoid CXFS Kernel Heartbeat Issues on Large Systems” in Chapter 2.

The following may cause the system to hang:

If you cannot access a filesystem, check the following:

If the log files are consuming too much disk space, you should rotate them; see “Log File Management” in Chapter 12. You may also want to consider choosing a less-verbose log level; see the following:

The Membership delivered messages in the system log file include a bitmask with a bit set for the cell IDs of nodes that are members of the new CXFS membership. The Membership delivered messages are followed by one or more messages starting with Cell(age): that print the individual cell IDs and the ages of their membership. In the following example, 704 is a 64-bit hexadecimal bitmask of cells included in the membership and cell 2 has been in the last 4 CXFS memberships:

[10798.974447] Cell 10 (cxfsxe14) joined the membership
[10798.989690] Membership delivered for cells
<0:0:0:0:0:0:0:704>
[10799.011084] Cell(age): 2(4) 8(7) 9(17) 10(1) 

If the Membership delivered messages are appearing frequently in the system log file, it may indicate a network problem:

See Chapter 8, “Postinstallation Steps”.

The following message indicates a problem (output lines wrapped here for readability):

ALERT: I/O error in filesystem ("/mnt") metadata dev 0xbd block 0x41df03 ("xlog_iodone")
ALERT:     b_error 0 b_bcount 32768 b_resid 0
NOTICE: xfs_force_shutdown(/mnt,0x2) called from line 966 of file ../fs/xfs/xfs_log.c. 
  Return address = 0xc0000000008626e8
ALERT: I/O Error Detected.  Shutting down filesystem: /mnt
ALERT: Please umount the filesystem, and rectify the problem(s)

You can fix this problem using xfs_repair only if there is no metadata in the XFS log. See “Forced Filesystem Shutdown Messages and XFS File Corruption”, for the appropriate procedure.

I/O errors can also appear if the node is unable to access the storage. This can happen for several reasons:

If you are unable to raise the fence on a node, it may be that the switch ports are unable to determine the WWPN. See “Hardware Changes and I/O Fencing” in Chapter 12.

If you have defined filesystems and then rename your cluster (by deleting the old cluster and defining a new cluster), CXFS will not be able to mount the existing filesystems. This happens because the clustered XVM volume on which your CXFS filesystem resides is not accessible to the new cluster, and the volumes are therefore considered as foreign.

In order to mount the filesystem on the new cluster, you must use the XVM steal command to bring the clustered XVM volume into the domain of the new cluster. For more information, see the XVM Volume Manager Administrator Guide.

The clconfd and cxfs_client daemons set the client_timeout value. The value depends on the order in which filesystems are mounted on the various nodes and adapts to help ensure that all filesystems get mounted in a timely manner. The value may differ among nodes and has no effect on the filesystem operation after it is mounted.

---

Caution: You should not attempt to change the client_timeout value. Improperly setting the values for client_timeout and retry could cause the mount command to keep waiting for a server and could delay the availability of the CXFS filesystems.

---

The retry value is forced to be 0 and you cannot change it.

On most platforms, the cxfs_client software automatically detects the world wide port names (WWPNs) of any supported host bus adapters (HBAs) in the system that are connected to a switch that is configured in the cluster database. These HBAs will then be available for fencing.

However, if no WWPNs are detected, there will be messages logged to the cxfs_client file.

If no WWPNs are detected, you can manually specify the WWPNs in the /etc/fencing.conf fencing file for the Linux platform. This method does not work if the WWPNs are partially discovered.

The fencing file enumerates the worldwide port name for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for the HBA WWPN as a 64-bit hexadecimal number.

---

Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric.

---

If used, the fencing file must contain a simple list of WWPNs, one per line.

If you use the fencing file, you must update it whenever the HBA configuration changes, including the replacement of an HBA.

Do the following:

After a filesystem has been defined in CXFS, running mkfs on it (or using “Make Filesystems with the GUI” in Chapter 10) will cause XFS internal errors to appear in the system log file. For example (line breaks added for readability):

Aug 17 09:25:52 1A:yokohama-mds1 unix: ALERT: Filesystem "(NULL)": XFS internal error
xfs_mount_validate_sb(4) at line 237 of file ../fs/xfs/xfs_mount.c.
Caller 0xc000000000326ef4

Aug 17 09:14:52 6X:yokohama-mds1 clconfd[360]: < E clconf 11> CI_FAILURE, fsinfo_update(/dev/cxvm/work)
kernel returned 1010 (Filesystem is corrupted)

To avoid these errors, run mkfs before defining the filesystem in CXFS or delete the CXFS filesystem before running mkfs.

If you have multiple metadata servers in the cluster but only one potential metadata server defined for a given filesystem and that server goes down, the now server-less filesystem goes into a shutdown state. Although the clients maintain membership in the cluster, they will not mount the filesystem automatically when the potential metadata server comes back up. You must manually unmount the filesystem.

If there had been only one potential metadata server in the cluster, the filesystem's clients would have lost membership and gone through a forced shutdown, which automatically unmounts the filesystems.

Forced filesystem shutdown messages do not necessarily imply that xfs_repair should be run. Following is an example of a message that does indicate an XFS file corruption:

XFS read error in file system metadata block 106412416

When a filesystem is forcibly shut down, the log is not empty -- it contains valuable metadata. You must replay it by mounting the filesystem. The log is only empty if the filesystem is unmounted cleanly (that is, not a forced CXFS shutdown, not a crash). You can use the following command line to see an example of the transactions captured in the log file:

xfs_logprint -t device

If you run xfs_repair before mounting the filesystem, xfs_repair will delete all of this valuable metadata.

You should run xfs_ncheck and capture the output to a file before running xfs_repair. If running xfs_repair results in files being placed in the lost+found directory, the saved output from xfs_ncheck may help you to identify the original names of the files.

---

Caution: Always contact SGI technical support before using xfs_repair on CXFS filesystems. See“Repair Filesystems with Care” in Chapter 2.

---

If you think you have a filesystem with real corruption, do the following from an Linux node with XFS commands RPM (sgi-xfsprogs ) installed:

This section discusses the following IPMI issues:

BMC Does Not Respond to a ping Command

If the baseboard management controller (BMC) does not respond to a ping(8) command from a remote node, verify that the BMC has a valid IP address assigned. See step 4 in Appendix C, “BMC System Controller”.

---

Note: The BMC will not respond to the ping command when issued from the local node (the node containing the BMC).

---

Sending clconfd a SIGTERM signal, the default signal sent by the kill(1) command, will cause the clconfd process to terminate. When the clconfd process terminates on a SIGTERM signal, it is not restarted by cmond and the node will remain in the CXFS cluster membership. All filesystem activity will continue without interruption. However, if clconfd is not running on one or more server-capable administration nodes in the cluster, configuration changes cannot be made in the cluster and CXFS recovery may hang, preventing nodes from joining the cluster membership.

The clconfd process will not run if there is a licensing problem. See “clconfd Daemon Death”.

If file access is slow, it could be due to one of the following problems:

There is only a single active admin login permitted to Fibre Channel, SAS, or InfiniBand switches that are properly configured for CXFS. If there is a defunct admin login remaining, CXFS may be unable to access the switch.

To solve this problem, do the following:

Rebooting the metadata server without first shutting down CXFS services can cause the metadata server to panic. You must use the proper procedures for shutting down nodes. See “Shut Down Nodes Unobtrusively” in Chapter 2.

The dynamic TCP port assignment used by the fs2d daemon may result in a collision with another service that is already using a fixed port number.

One example that has been observed is Samba hanging during startup when trying to query the CUPS printing system for printer information. Other applications that expect to connect on a well-known port may be affected.

To see which ports fs2d is using, use the following command:

# lsof -i TCP | grep fs2d

If you encounter this problem, one option is to restart fs2d with a SIGHUP signal so that it will move to another port:

# /usr/bin/killall -HUP fs2d

You may have to repeat this command until fs2d is assigned to a port that is not in contention.

If you restart rpcbind without the -w option to cause a warm start, the fs2d daemon will not be able to communicate with other CXFS daemons, and the cluster will not function normally. In this case, you should force a restart of the fs2d daemon so that it can continue to communicate with the various CXFS daemons:

# killall -9 fs2d

The Brocade telnet login session can become hung for Brocade switches running firmware version V5* and later. To clear the situation, do the following:

If a CXFS client fails or exits the cluster during the metadata server relocation process, the relocation process and the client recovery are likely to hang. This prevents any clients, including the failed client, from joining the cluster.

Once in this state, it may be possible to resolve the deadlock by resetting or power-cycling the fs2d quorum master. See “Determine the Quorum Master”.

NFSv4-exported filesystems do not display properly in df output on Linux nodes. This is an NFS issue that exists independent of CXFS.

If mkinitrd fails, it may be because a new kernel module was installed but the sgi-xfsprogs RPM was not installed. To resolve this problem, install the sgi-xfsprogs RPM and rerun mkinitrd.

CXFS token prefetch and range tokens are designed as optimizations for applications using CXFS filesystems on a CXFS client. However, under some workloads, may cause stability issues. If directed to do so by SGI Support, you can use the cell_tkm_feature_disable system tunable parameter to disable these features on Linux and Mac OS X clients.

---

Note: You should modify cell_tkm_feature_disable only if directed to do so by SGI Support.

---

For more information, see “cell_tkm_feature_disable” in Appendix D and the Linux and Mac OS X chapters of CXFS 7 Client-Only Guide for SGI InfiniteStorage.

If you encounter membership issues after installing or upgrading licenses, you may need to restart the fs2d daemon. See “Restarting fs2d After Installing or Upgrading Licenses” in Chapter 5.

If an XVM volume is not functioning on only one node, it may be that there is a mismatch of required RPMs on that node. For XVM to function properly, a node must have matching versions of the following RPMs:

If there is an inconsistency in the RPM levels, the XVM volume may function normally on the CXFS metadata server and other client-only nodes. However, on the node with a mismatch of RPM versions, XVM will report the volume as problematic (as offline, as inaccessible, and/or with no physical connection) despite the fact that the SCSI device ( /dev/sd*) can be read without problems. For example:

xvm:cluster> show *
phys/zj                           0 cluster,inaccessible
slice/zjs0                366878464 offline,open,inaccessible
slice/zjs1                366878464 offline,inaccessible
slice/zjs2                366878464 offline,inaccessible
slice/zjs3                366878464 offline,inaccessible
subvol/zjs0/data          366878464 offline,pieceoffline,open,inaccessible
subvol/zjs1/data          366878464 offline,pieceoffline,inaccessible
subvol/zjs2/data          366878464 offline,pieceoffline,inaccessible
subvol/zjs3/data          366878464 offline,pieceoffline,inaccessible
vol/zjs0                          0 offline,open,no physical connection,inaccessible
vol/zjs1                          0 offline,no physical connection,inaccessible
vol/zjs2                          0 offline,no physical connection,inaccessible
vol/zjs3                          0 offline,no physical connection,inaccessible

If a mismatch is the cause, a message similar to one of the following should appear in the system log:

XVM api version 24 doesn't match kernel api version 23: kernel/user api

    pm api version mismatch: user:7, kernel:8

To resolve the problem, upgrade the out-of-date client-only node with the appropriate RPMs.

Many messages are normal and do not indicate a problem. Following is a subset of normal messages:

The following are common error and warning messages displayed by cxfs-client. For more information, see “Validating the Cluster Configuration with cxfs-config ” in Chapter 13.

enabled machine hostname has no NICs matching any net

There are failover networks defined, but the network interfaces for hostname are not members of the defined networks. See:

one or more machines have fencing enabled, but no switches are defined

CXFS requires either system reset or I/O fencing for all nodes. Fencing requires a switch. See “Create or Modify a Node with cxfs_admin” in Chapter 11.

server hostname fail policy must not contain "Shutdown" for cluster 
      with even number of enabled servers and no tiebreaker

If there are an even number of server-capable administration nodes and there is no tiebreaker set, the fail policy must not contain the shutdown option because there is no notification that a shutdown has occurred. See:

tiebreaker must not be set in cluster with exactly two enabled nodes  both server capable

If exactly two server-capable administration nodes are configured and there are no client-only nodes, neither server-capable administration node should be set as the tiebreaker. See “Use a Client-Only Tiebreaker” in Chapter 2.

If you try to relocate a filesystem and see an error similar to the following cxfs_admin example, it means that relocation has not been enabled:

Error returned from server: feature not enabled (12)
Command "relocate slice1C server=server1" failed during commit: feature not enabled

To allow the relocation to occur, you must enable relocation as specified in “Relocation” in Chapter 1.

If you see an error message such as the following when trying to shut down a node, it means that the node which is the active metadata server for one or more filesystems and was unable to unmount the filesystems, and therefore was unable to shutdown gracefully:

Aug 31 10:06:18 cc-xe kernel: Filesystem "xvm-2": relocation from this host is disabled

Also see “Shutdown Failure”.

If a node is unable to shutdown gracefully, it may be because it was unable to unmount the filesystems for which it is the active metadata server. See “Relocation Error”.

If you see messages such as the following on the console or in a message log, it means that the Fibre Channel, SAS, or InfiniBand switch is misconfigured:

controller disable is not supported on loop

CXFS fencing recovery operations do not support loop mode. Verify that all switches are configured correctly. See the switch documentation for configuration information.

The following messages may be logged by CMS:

CMS excluded cells <0:0:0:0:c000000000000000:0:0:704> with incomplete connectivity

Generated when CMS delivers a membership that excluded some new cells that had not established connections with enough cells yet to be admitted. 704 is a binary bitmask of excluded cells.

CMS calculation limited to last membership:configuration change incomplete on cells <0:0:0:0:0:0:0:704>

Generated when the leader is attempting to make a configuration change current (that is, actually use the change on all nodes), but some cells in the cluster have not yet gotten the configuration change staged (uploaded and ready to be made current). 704 is a binary bitmask of cells that do not yet have the change in their configuration. Changes make their way through the cluster asynchronously, so this situation is expected. It can take a few attempts by the CMS leader before all nodes have the change staged. As long as this situation resolves eventually, there is no problem.

CMS calculation limited to last membership:recovery incomplete

Generated when new members were disallowed due to recovery from the last cell failure that is still being processed.

Cell XXX is tardy

Generated when cell XXX has not reported in to the cluster membership leader during a membership change, resulting in a timeout. This situation will cause the node to be removed from the cluster membership and can result in a shutdown of the entire cluster if quorum is lost.

If the clconfd daemon exits immediately after it starts up, it usually means that the CXFS license key has not been properly installed. Check the end of the clconfd log file (/var/log/cxfs/clconfd_ nodename) for error messages. For information about licensing error messages, see “License Key Error”.

You must properly install the license keys before you can use CXFS. See Chapter 5, “CXFS Licensing”.

The following example system log file message indicates an oversubscribed system:

ALERT: inetd [164] - out of logical swap space during fork while
allocating uarea - see swap(1M)
Availsmem 8207 availrmem 427 rlx freemem 10, real freemem 9

See “Use System Capacity Wisely” in Chapter 2.

Daemons could also be leaking memory in this case. You may need to restart them:

The following message in the system log file indicates a kernel-triggered revocation of CXFS membership:

Membership lost - withdrawing from cluster

You must allow CXFS membership for the local node in this situation. See “Allow Membership of the Local Node with the GUI” in Chapter 10 or “Enable a Node with cxfs_admin” in Chapter 11.

You will see the following error if you try to install CXFS on a server-capable administration node without a valid license key already in place:

Preparing...                ###########################################
[100%]
   1:cxfs_cluster           ###########################################
[100%]
cxfs                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
cluster_cx-exitop: Added CXFS keys to /var/cluster/cdb/cdb.db
cluster_cx-exitop: Added CXFS administration access keys to
/var/cluster/cdb/cdb.db
cxfs license check failed - use '/usr/cluster/bin/cxfslicense -d' for
details

         * * * * * * * * * * I M P O R T A N T * * * * * * * * * * * * *

        CXFS is not properly licensed for this host.  Run
                '/usr/cluster/bin/cxfslicense -d'
        for more detailed license information.
        After fixing the license, please run
                '/bin/true; /etc/init.d/cxfs_cluster restart'.

cluster_cx-exitop: success

If you see the following message in the /var/log/cxfs/clconfd_nodename logfile, it means that the CXFS license key was not properly installed:

CXFS not properly licensed for this host.  Run
               '/usr/cluster/bin/cxfslicense -d'
       for detailed failure information. 

If you do not have the CXFS license key properly installed, you will see an error on the console when trying to run CXFS. For example:

Cluster services:CXFS not properly licensed for this host.  Run
        '/usr/cluster/bin/cxfslicense -d'
for detailed failure information.  After fixing the
license, please run '/etc/init.d/cxfs_cluster restart'.

An error such as the following example will appear in the system log file:

Mar  4 12:58:05 6X:typhoon-q32 crsd[533]: <<CI> N crs 0> Crsd restarted.
Mar  4 12:58:05 6X:typhoon-q32 clconfd[537]: <<CI> N clconf 0> 
Mar  4 12:58:05 5B:typhoon-q32 CLCONFD failed the CXFS license check.Use the 
Mar  4 12:58:05 5B:typhoon-q32    '/usr/cluster/bin/cxfslicense -d'
Mar  4 12:58:05 5B:typhoon-q32 command to diagnose the license problem.

If the clconfd daemon dies right after it starts up, this error may be present.

An error such as the following example will appear in the SYSLOG file (line breaks added here for readability):

Jan 25 10:24:03 ncc1701:Jan 25 10:24:03 cxfs_client:
cis_main FATAL: cxfs_client failed the CXFS license check.
Use the cxfslicense command to diagnose the license problem

Message similar to the following will appear in the client-log file:

On a server-capable administration node, the error will appear in the clconfd log.

You must properly install the license key before you can use CXFS. See Chapter 5, “CXFS Licensing”.

If you have conflicting cluster ID numbers at your site, you will see errors such as the following:

WARNING: mtcp ignoring  alive message from 1 with wrong ip addr 128.162.89.34
WARNING: mtcp ignoring  alive message from 0 with wrong ip addr 128.162.89.33

A cluster ID number must be unique. This error can occur if you redefine the cluster configuration and start CXFS services while some nodes have stale information from a previous configuration.

To solve this problem, make the cluster ID numbers unique. First try the steps in “Eliminate a Residual Cluster”. If that does not work, reboot the nodes that have stale information. Stale nodes will complain about all of the nodes, but the up-to-date nodes will complain only about the stale nodes. The /var/log/cxfs/clconfd_ log file on the stale nodes will also show error messages about SGI_CMS_CONFIG_ID failures.

If there are too many error messages to recognize the stale nodes, reboot every node.

CXFS logs both normal operations and critical errors to the system log file, as well as to individual log files for each log group.

The system log file on server-capable administration nodes is in /var/log/messages.

In general, errors in the system log file take the following form:

timestamp priority_&_facility : hostname process[ID]: <internal_info> CODE message_text 

For example:

Sep  7 11:12:59 6X:cxfs0 cli[5830]: < E clconf 0> CI_IPCERR_NOSERVER, clconf
ipc: ipcclnt_connect() failed, file /var/cluster/comm/clconfd-ipc_cxfs0

Table 15-2 breaks down the parts of the preceding message.

The following sections present only the message identifiers and text:

fs2d  - CIS config: multi:IPaddress:5449, delay=1s, incarnation=0x41cc5f0bcdb1c235, flags=

error reading kernel XVM cluster mirror status. Check if XVM module is
started.

CXFS maintains logs for each of the CXFS daemons. For information about customizing these logs, see “Set Log Configuration with the GUI” in Chapter 10.

Log file messages take the following form:

daemon_log timestamp internal_process: message_text

For example:

cad_log:Thu Sep  2 17:25:06.092  cclconf_poll_clconfd: clconf_poll failed with error CI_IPCERR_NOPULSE

Table 15-3, shows the parts in the preceding message.

This section discusses the following:

Thu Jun 3 16:20:45.431 cxfsopus1.example.com cbe_fs2 - cbe_create_node: cannot create new node (RPC error = 9)
  libcdb       - cdb_create_node: error 9 creating child of node 0x60000000000135c0 with subkey "ifd1"

This error means that some nodes have not been created in the cluster database. Error 9 usually means that fs2d is has encountered an internal error while creating that node. To fix the problem, make sure that fs2d is not running on any server-capable administration node and rerun cdbreinit.

Following are common cxfs_admin errors.

Connecting to the local CXFS server...
receiving conflicting bootstrap packets from cluster(s) - cannot identify
server to connect to
gave up trying to connect to server
FATAL: exiting on fatal error

The cxfs_admin command can see multiple clusters. Reconfigure your network so that each cluster's private network subnet is independent of the private network subnet of other clusters. If you have multiple clusters connected to the same public network, use the -i option to identify the cluster name. See “Accessing the Correct Cluster at a Multiple-Cluster Site” in Chapter 11.

Connecting to the CXFS server for the "mycluster" cluster...
Error returned from server: authorization error (8)
Inappropriate privileges to connect to the CXFS server

The host can see the cluster, but does not have permission to connect to it. Use cxfs_admin on another node that does have permission and use the access command to give permission to the first node.

Connecting to the CXFS server for the "mycluster" cluster...
Error returned from server: permissions error (9)
Insufficient privileges to acquire the administration lock

The host only has monitoring privileges and no administration privileges. Use the permission=admin attribute with the access command to grant the host administration rights, or use -r on the cxfs_admin command line.

Connecting to the CXFS server for the "mycluster" cluster...
not receiving bootstrap packets from any cluster - cannot identify server to connect to
gave up trying to connect to server
FATAL: exiting on fatal error

The host is not on the CXFS metadata private network and has not been granted explicit access to the cluster. Grant the host access by using the access command from a server-capable administration node or another host with admin access to the cluster.

transient bootstrap state for cluster clustername detected

This message usually indicates that there are two clusters with the same name running on the network. You must rename one of the clusters.

The following error indicates that one of the LUNs in this volume is inaccessible. A GPT-labeled LUN in the volume may cause this if GPT labels are not supported on the system:

# /sbin/mount -t cxfs -o 'client_timeout=30s,retry=0,server_list=(server1,server2)' \
/dev/cxvm/stripe93 /mnt/stripe93
 cxfs.util get_subvol_stripe: open(/dev/rcxvm/stripe93) returned -1, errno 19 (Operation not supported by device)
 cxfs.util get_subvol_stripe: Some of the volumes needed for /dev/rcxvm/stripe93 may have a main path that 
   runs through a controller to which this machine is not connected.
 cxfs.util set_xfs_args: get_subvol_stripe failed
 cxfs.util mount_main: set_xfs_args failed

If a node is not allowed to be configured automatically by the cxfs_admin command, errors such as the following will appear in the cxfs_client log:

Aug 06 10:36:51 cxfs_client: cis_cdb_go ERROR: Error returned from server: authorization
error (8)
Aug 06 10:36:51 cxfs_client: cis_autoconf client is not in an autoconf "allowed" policy

For more information about automatic configuration, see “Automatically Configure a Client-Only Node” in Chapter 11.

The cxfs_admin command will also display errors if the node running cxfs_admin does not have admin or monitor access to the cluster. See “Setting cxfs_admin Access Permissions” in Chapter 11.

The following message indicates that the system is unable to open UDP socket connection to another system (in this case, cell ID 4) because the connection attempt is refused:

mesg_connect:cell 4:channel 0:transport 2:connect error -111

Frequently repeated occurrences of this message may indicate an incorrect network configuration or may require reinstalling the CXFS software on either the CXFS client-only node or the server-capable administration node. This message may also occur occasionally when a node is started or shut down.

The following error is generated when the cluster contains nodes running incompatible versions of CXFS software, such as cluster with nodes running CXFS 7.0 (ISSP 3.0) and CXFS 6.6 (ISSP 2.6):

[  552.680802] Cell 3 (pg-5) transport [from 192.168.14.163:5450 to
192.168.14.54:5450] connection refused: remote version is too old.

All server-capable administration nodes in the cluster must run the same version of CXFS except during a rolling upgrade. The following types of errors indicate that the server-capable administration nodes are running different versions of CXFS software:

CXFS cell C (name) mount refused: node is downrev (tag X, has N, need M)

CXFS cell C (name) relocation refused: node is downrev (tag X, has N, need M)

Cell C (nodename) is downrev and will not be able to connect

An operation may be forbidden because of a version mismatch between the metadata server and the other node involved. The following operations may fail:

For example, the following message indicates that a client-only node tried to mount a filesystem served by a metadata server that is running an older version of CXFS:

CXFS cell 0 (xo-xe1) mount refused: node is downrev (tag 30, has -1, need 0)

In another example, the following message that appears on a client-only node (with a cell ID of 5) indicates that server-capable administration node cxfsxe6 (with a cell ID of 9) is running an older CXFS release:

Cell 9 (cxfsxe6) is downrev and will not be able to connect

In this case, the following message will also appear at 1-second intervals on cxfsxe6:

mesg_connect:cell 5:channel 0:transport 2:connect error -111

This combination of messages indicates that the client-only node is running a version of CXFS from ISSP 2.x and cxfsxe6 is a server-capable administration node running a later version of CXFS from ISSP 1.x, which is an unsupported configuration.

During a rolling upgrade, relocation of a filesystem between server-capable administration nodes running different versions of CXFS may be disabled. Recovery rather than relocation is used to move filesystems to an upgraded server-capable node. See “CXFS Release Versions and Rolling Upgrades” in Chapter 12.

If you see messages like the following on the console or in the system log file, turn off CXFS extents deltas:

EXTENT: a8000002c9e60600 bno 10 not at boundary (got = 9+5)

This is best done on the metadata server by setting the fs.cxfs.cxfs_extents_delta system tunable parameter to 0. A reboot is not required. For more information, see “cxfs_extents_delta” in Appendix D.

If you see errors like the following, it means that the server-capable administration node was not rebooted after installing GRIOv2:

# /usr/sbin/grioadmin -sv
print_streams: error: Cannot get GRIO server info: Resource temporarily unavailable
grioadmin: print_streams: error: grio_get_streams: Resource temporarily unavailable

The following are examples of errors that can be produced by the xvm foswitch command, which changes the path used to access a physical disk:

mds1-hd kernel: foswitch error i/o error on cell 19

The test I/O on cell 19 failed when trying the new path.

mds1-hd kernel: foswitch error failover: no device of specified affinity on cell 9

There is no a path with the desired affinity on cell 9 . For more information, see the failover2.conf file on the node with the specified cell ID.

mds1-hd kernel: foswitch error failover: potentially recoverable error, try again on cell 0

An foswitch command was attempted while a failover was happening.

For more information, see XVM Volume Manager Administrator Guide.

Multiple messages of the following type indicate that the port on the switch is not fenced but the Fiber Channel connection with the node is gone:

<I0 clconfd clconf 25808:10 clconfd_fence.c:245> sw
msdsan2, port 5, host dpox01 : leaving transient state (timeout => no
connection), clearing cdb entries and lowering fences on this port

You should check the connection between the nodes and the switches, and verify that the switches are running properly.

The following error message when trying to stop the CXFS filesystem service (cxfs) may indicate that filesystems cannot be unmounted:

# /etc/init.d/cxfs stop
Stopping CXFS cluster services...
Stopping clconfd:
                                                                        done
CXFS Cluster Admin Shutdown:
CXFS Cluster Admin Shutdown failed
                                                                        failed
/etc/init.d/cxfs: line 333:  2200 Killed          $CAP_CXFS "${BIN}/cxfs_shutdown -f -r 0" > /dev/null 2>&1

MDS:
...

This can be the result if the node has already been removed from the cluster (despite the shutdown failure message).

The following message indicates that a CXFS node on the CXFS private network is running an older, incompatible version of CXFS software:

mtcp ignoring improperly sized alive message from node_IP_address:nnnnn (got 16 want 72)

You should disable or upgrade the CXFS software software on the indicated node. If the node belongs to a different cluster, the private networks for the two clusters should be segregated. Running multiple clusters on a shared private network is not a supported configuration.

This message may also occur if there is more than one cluster using the public network as a backup private network. This is a valid configuration. In this case the message frequency can be modified or the messages disabled using the mtcp_hb_warn_period system tunable parameter. See “mtcp_hb_warn_period” in Appendix D.

The following error indicates that the XVM server is waiting for a reply from the client with the identified cell number, but the client has not responded:

mds kernel: remote_clients_verify: cell NN timed out

XVM operations in the cluster will be blocked until the client responds. If you seen the message, review the client's system log for I/O errors and try to execute the following command on the client:

client# xvm show -v phys

Most likely, this xvm command will hang. If so, trigger an NMI or otherwise panic the system in order to generate a kernel dump.

It is normal for the following kernel log message to appear occasionally on CXFS server-capable administration nodes, where X and Y are replaced by the cell IDs of the actual nodes involved:

XVM client X unable to lock bootconfig, held by cell Y

However, a problem is indicated when the message appears continually. To resolve the problem, reset cell Y.

If CXFS services to do not restart after a reboot, it may be that the node was marked as INACTIVE in the cluster database using the Stop CXFS Services function of the GUI or a disable node:nodename function of cxfs_admin. In this case, issuing a service cxfs_cluster start or service cxfs start will not restart the services.

You must manually start CXFS services. If you use the GUI to restart the services, or enable with cxfs_admin, the configuration will be set so that future reboots will also restart CXFS services.

For information, see:

To clear the cluster database on all of the server-capable administration nodes, do the following, completing each step on each server-capable administration node before moving to the next step:

---

Caution: This procedure deletes all configuration information.

---

See “Eliminate a Residual Cluster” to get rid of possible stale cluster configuration in the kernel. If needed, reboot the nodes.

The following are situations that may require rebooting:

Enter the following individually on every node to reboot the cluster (other than Windows, which uses a different reboot mechanism):

# reboot

If you want CXFS services to restart whenever the node is rebooted, use the CXFS GUI to start CXFS services or cxfs_admin to enable the node. For information, see:

For information about client-only nodes, see the CXFS 7 Client-Only Guide for SGI InfiniteStorage.

The cxfs_cluster argument to chkconfig controls the other cluster administration daemons and the replicated cluster database. If turned off, the database daemons will not be started at the next reboot and the local copy of the database will not be updated if you make changes to the cluster configuration on the other nodes. This could cause problems later, especially if a majority of nodes are not running the database daemons.

If the cluster daemons are causing serious trouble and prevent the machine from booting, you can recover the node by booting in single-user mode, turning the argument off and booting in multiuser mode.

For example:

ELILO boot: linux single
Uncompressing Linux... done
 ...
Skipped services in runlevel S:                   splash
Give root password for login: *****

(none)# /sbin/chkconfig grio2 off (if running GRIOv2)
(none)# /sbin/chkconfig cxfs off
(none)# /sbin/chkconfig cxfs_cluster off
(none)# reboot

For more information, see “chkconfig Arguments” in Chapter 12.

There are specific issues with recovering a cluster with two server-capable administration nodes. Suppose the following sequence of events:

To allow node2 to form a cluster by itself, you must do the following:

The cluster will attempt to communicate with the node1 because it is still configured in the cluster, even though it is down. Therefore, it may take some time for the CXFS kernel membership to form and for filesystems to mount.

---

Caution: When the cluster administration daemons are stopped, the node will not receive database updates and will not update the kernel configuration. This can have very unpleasant side effects. Under most circumstances, the administration daemons should remain running at all times. Use these commands only as directed.

---

The commands to stop and restart cluster administration daemons depend upon the platform.

To stop and restart cluster administration daemons, enter the following:

These commands affect the cluster administration daemons only.

See also “Restarting CXFS Services”. For general information about the daemons, see “Kernel Threads” in Appendix A.

See Chapter 13, “Cluster Database Management”. If a node will not join cluster membership, you must reboot it.

Verification of multicast connectivity requires multicast ping on each node in the cluster (other than Windows nodes).

On Linux nodes, you must enable multicast ping by using one of the following methods (the permanent method will not take affect until after a reboot):

To verify general connectivity in a multicast environment, you can execute a ping command on the 224.0.0.1 IP address; 224.0.0.1 is the address for all systems on this subnet multicast.

To verify the CXFS kernel heartbeat, use the 224.0.0.250 IP address. (SGI uses 224.0.0.250 by default; you may need to explicitly set this if you are having problems with name resolution involving cluster_mcast, see below).

---

Note: A node is capable of responding only when the administration daemons (fs2d, cmond, cad, and crsd) or the cxfs_client daemon is running.

---

For example, to see the response for two packets sent from IP address 163.154.17.49 to the multicast address for CXFS kernel heartbeat and ignore loopback, enter the following:

nodeA# ping -c 2 -I 163.154.17.49 -L 224.0.0.250
PING 224.0.0.250 (224.0.0.250): 56 data bytes
64 bytes from 163.154.17.140: icmp_seq=0 ttl=64 time=1.146 ms
64 bytes from 163.154.17.55: icmp_seq=0 DUP! ttl=255 time=1.460 ms
64 bytes from 163.154.17.52: icmp_seq=0 DUP! ttl=255 time=4.607 ms
64 bytes from 163.154.17.50: icmp_seq=0 DUP! ttl=255 time=4.942 ms
64 bytes from 163.154.17.140: icmp_seq=1 ttl=64 time=2.692 ms

----224.0.0.250 PING Statistics----
2 packets transmitted, 2 packets received, +3 duplicates, 0.0% packet
loss
round-trip min/avg/max = 1.146/2.969/4.942 ms

The above output indicates that there is a response from the following addresses:

163.154.17.140
163.154.17.55
163.154.17.52
163.154.17.50

To override the default address, you can use the -c and -m options on client-only nodes; for more information, see the cxfs_client man page.

To override or explicitly set the CXFS multicast address, do the following:

To power-cycle a node, see “Control and Contact a Node with cxfs_admin” in Chapter 11.

To reset a node, see “Control and Contact a Node with cxfs_admin” in Chapter 11.

In the case of an abrupt power outage, you can bring up CXFS without mounting the previously mounted filesystems by doing the following:

If you encounter problems and have an SGI support contract, see:

https://support.sgi.com

If you need further assistance, contact SGI Support.