This section provides an overview of the tools required to troubleshoot CXFS:

---

Caution: Many of the commands listed are beyond the scope of this book and are provided here for quick reference only. See the other guides and man pages referenced for complete information before using these commands.

---

lcrash Kernel Status Tools

:Use the appropriate version of lcrash and load the CXFS kerntypes:

# lcrash -x /boot/sgi-cxfs-kerntypes-kernelversion-architecturetype

---

Note: Do not use the version of lcrash that is shipped with SLES. Use the version of lcrash that is available from Supportfolio.

---

Gather Cluster Configuration with cxfsdump

Before reporting a problem to SGI, you should use the cxfsdump command to gather configuration information about the CXFS cluster, such as network interfaces, CXFS registry information, I/O, and cluster database contents. This will allow SGI support to solve the problem more quickly.

---

Note: In cluster mode (the default when run on a server-capable administration node), the cxfsdump command requires rsh/ssh and rcp/ scp access across all nodes in the cluster. You can use the -secure option to use secure remote connections.

---

You should run cxfsdump from a server-capable administration node in the cluster:

server-admin# /usr/cluster/bin/cxfsdump

The output will be placed in a file in the directory /var/cluster/cxfsdump-data directory on the server-capable administration node on which the cxfsdump command was run. The cxfsdump command will report the name and location of the file when it is finished.

To gather information about just the local node, use the cxfsdump -local option.

You can configure the location of the dump by selecting the directory from a browse for folder dialog or type in the path in the edit field.

For more information about client-only nodes, see CXFS 5 Client-Only Guide for SGI InfiniteStorage.

When you encounter a problem, identify the cluster status by answering the following questions:

Before you start configuring another new cluster, make sure no nodes are still in a CXFS membership from a previous cluster. Enter the following to check for a cmsd kernel thread:

server-admin# ps -ef | grep cmsd

If the output shows a cmsd kernel thread, perform a forced CXFS shutdown by entering the following:

server-admin# service cxfs stop

Then check for a cmsd kernel thread again.

After waiting a few moments, if the cmsd kernel thread still exists, you must reboot the machine or leave it out of the new cluster definition. It will not be able to join a new cluster in this state and it may prevent the rest of the cluster from forming a new CXFS membership.

To determine if a node is fenced, log in to a server-capable administration node and use the cxfs_admin status command or the hafence(1M) command.

The following messages are logged when fencing changes:

Raising fence on cell cellID (nodename)

Lowering fence on cell cellID (nodename)

The quorum master is the server-capable administration node with the lowest node ID number (not cell ID) . To confirm which node is the quorum master, look in the var/cluster/ha/log/fs2d_log file for the most recent quorum message, such as the following example, which shows that node3 is the quorum master:

Tue Jul  7 12:52:41.490 cxfsopus4 fs2d  - Checking quorum with 4 members for any unknown members.
Tue Jul  7 12:52:41.490 cxfsopus4 fs2d  -  All quorum member machines known, master is node3

To locate the problem, do the following:

Brocade switch problems can cause CXFS to behave abnormally. For easier troubleshooting, use the syslogdipadd function on the switch to redirect its syslogd information to up to six potential metadata servers in the cluster. SGI recommends logging to at least two potential metadata servers on which you troubleshoot issues and look for error messages. The syslogd information is the same as that given by errshow command on the switch.

For example, on each switch, define the metadata server nodes MDS1 and MDS2 to which the switch can redirect its syslogd output:

switch:admin > syslogdipadd ipaddress_MDS1
switch:admin > syslogdipadd ipaddress_MDS2

The entries from the switch can be sorted because they are prefixed by the switch name, which is standard syslogd behavior.

The following messages indicate that a client has lost membership (line breaks added here for readability):

Mar 15 10:55:35 5A:mvcxfs2 kernel: Error -1 reading mesg header channel 0 cell 4 (mvcxfs17) 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:35 4A:mvcxfs2 kernel: Error receiving messages from cell 4 (mvcxfs17) tcpchannel 0 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:36 5A:mvcxfs2 kernel: Error -1 reading mesg header channel 1 cell 4 (mvcxfs17) 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:36 4A:mvcxfs2 kernel: Error receiving messages from cell 4 (mvcxfs17) tcpchannel 1 
  [priority 1 at 192.168.17.173 via 192.168.17.48]
Mar 15 10:55:36 5A:mvcxfs2 kernel: Error -1 reading mesg header channel 1 cell 4 (mvcxfs17) 
  [priority 2 at 163.154.17.173 via 163.154.17.48]
Mar 15 10:55:36 4A:mvcxfs2 kernel: Error receiving messages from cell 4 (mvcxfs17) tcpchannel 1 
  [priority 2 at 163.154.17.173 via 163.154.17.48]
Mar 15 10:55:36 4A:mvcxfs2 kernel: Transport failure cell 4 
  [priority 2 at 163.154.17.173 via 163.154.17.48] 0 of 2 interfaces up
Mar 15 10:55:36 6A:mvcxfs2 kernel: Heartbeat Monitor:Failure time-stamp 295789 ticks:Last heartbeat 
  time-stamp 289940 ticks:Time-stamp delta 5849 ticks (5 seconds):Heartbeat timeout 5120 ticks (5 seconds)

The Error receiving and Error reading messages indicate that the message channel went down. The last message, which includes the Heartbeat Monitor string, contains other strings that give a clues as to why the channel was disconnected. Table 15-1 lists all of the possible strings that may be included.

In the above example, the last message indicates that there is a heartbeat timeout because the string Heartbeat Monitor is included. The message also indicates that the error was detected at 295789 ticks (Failure time-stamp string) and that the configured timeout is 5120 ticks or 5 seconds (the Heartbeat timeout string). The delta is 5849 ticks or 5 seconds (the Time-stamp delta string), therefore it is a heartbeat timeout because the delta is greater than the configured heartbeat timeout.

If you are unable to raise the fence on a node, it may be that the switch ports are unable to determine the WWPN. See “Hardware Changes and I/O Fencing” in Chapter 12.

If you cannot access a filesystem, check the following:

If the log files are consuming too much disk space, you should rotate them; see “Log File Management” in Chapter 12. You may also want to consider choosing a less-verbose log level; see the following:

If you are unable to define a node, it may be that there are hostname resolution problems. See “Hostname Resolution and Network Configuration Rules” in Chapter 6.

The following may cause the system to hang:

If a node is detected in the system log file but it never receives a Membership delivered message, it is likely that there is a network problem.

See Chapter 8, “Postinstallation Steps”.

The Membership delivered messages in the system log file include a bitmask with a bit set for the cell IDs of nodes that are members of the new CXFS membership. The Membership delivered messages are followed by one or more messages starting with Cell(age): that print the individual cell IDs and the ages of their membership. 0xXXX is a binary bitmask of cells included in the membership. In the following example, cell 0 has been in the last 21 CXFS memberships:

NOTICE: Membership delivered for cells 0x3.
Cell(age): 0(21) 1(12)

If the Membership delivered messages are appearing frequently in the system log file, it may indicate a network problem:

See Chapter 8, “Postinstallation Steps”.

If you cannot log in to a server-capable administration node, you can use one of the following commands, assuming the node you are on is listed in the other nodes' .rhosts files:

rsh hostname ksh -i
rsh hostname csh -i

The following message indicates a problem (output lines wrapped here for readability):

ALERT: I/O error in filesystem ("/mnt") metadata dev 0xbd block 0x41df03 ("xlog_iodone")
ALERT:     b_error 0 b_bcount 32768 b_resid 0
NOTICE: xfs_force_shutdown(/mnt,0x2) called from line 966 of file ../fs/xfs/xfs_log.c. 
  Return address = 0xc0000000008626e8
ALERT: I/O Error Detected.  Shutting down filesystem: /mnt
ALERT: Please umount the filesystem, and rectify the problem(s)

You can fix this problem using xfs_repair only if there is no metadata in the XFS log. See “Forced Filesystem Shutdown Messages and XFS File Corruption”, for the appropriate procedure.

I/O errors can also appear if the node is unable to access the storage. This can happen for several reasons:

If you are unable to raise the fence on a node, it may be that the switch ports are unable to determine the WWPN. See “Hardware Changes and I/O Fencing” in Chapter 12.

If you have defined filesystems and then rename your cluster (by deleting the old cluster and defining a new cluster), CXFS will not be able to mount the existing filesystems. This happens because the clustered XVM volume on which your CXFS filesystem resides is not accessible to the new cluster, and the volumes are therefore considered as foreign.

In order to mount the filesystem on the new cluster, you must use the XVM steal command to bring the clustered XVM volume into the domain of the new cluster. For more information, see the XVM Volume Manager Administrator's Guide .

If you create new slices on a previously sliced disk that have the same starting blocks as slices already existing on the disk, and if the old slices had filesystems, then the GUI will display those old filesystems even though they may not be valid.

A client_timeout value is set by the clconfd and cxfs_client daemons. The value depends on the order in which filesystems are mounted on the various nodes. The value adapts to help ensure that all filesystems get mounted in a timely manner. The value has no effect on the filesystem operation after it is mounted.

The value for client_timeout may differ among nodes, and therefore having multiple values is not really a problem.

The retry value is forced to be 0 and you cannot change it.

---

Caution: You should not attempt to change the client_timeout value. Improperly setting the values for client_timeout and retry could cause the mount command to keep waiting for a server and could delay the availability of the CXFS filesystems.

---

On most platforms, the cxfs_client software automatically detects the world wide port names (WWPNs) of any supported host bus adapters (HBAs) in the system that are connected to a switch that is configured in the cluster database. These HBAs will then be available for fencing.

However, if no WWPNs are detected, there will be messages logged to the cxfs_client file.

If no WWPNs are detected, you can manually specify the WWPNs in the /etc/fencing.conf fencing file for the Linux platform. This method does not work if the WWPNs are partially discovered.

The fencing file enumerates the worldwide port name for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for the HBA WWPN as a 64-bit hexadecimal number.

---

Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric.

---

If used, the fencing file must contain a simple list of WWPNs, one per line.

If you use the fencing file, you must update it whenever the HBA configuration changes, including the replacement of an HBA.

Do the following:

After a filesystem has been defined in CXFS, running mkfs on it (or using “Make Filesystems with the GUI” in Chapter 10) will cause XFS internal errors to appear in the system log file. For example (line breaks added for readability):

Aug 17 09:25:52 1A:yokohama-mds1 unix: ALERT: Filesystem "(NULL)": XFS internal error
xfs_mount_validate_sb(4) at line 237 of file ../fs/xfs/xfs_mount.c.
Caller 0xc000000000326ef4

Aug 17 09:14:52 6X:yokohama-mds1 clconfd[360]: < E clconf 11> CI_FAILURE, fsinfo_update(/dev/cxvm/work)
kernel returned 1010 (Filesystem is corrupted)

To avoid these errors, run mkfs before defining the filesystem in CXFS, or delete the CXFS filesystem before running mkfs.

In Altix systems with multiple Ethernet interfaces, the default behavior of the operating system is to dynamically assign interface names (such as eth0, eth1, and so on) at boot time. Therefore, the physical interface associated with the eth0 device may change after a system reboot; if this occurs, it will cause a networking problem for CXFS. To avoid this problem, provide persistent device naming by using the /etc/sysconfig/networking/eth0_persist file to map specific Ethernet device names to specific MAC addresses. Adding lines of the format to the eth0_persist file:

ethN MAC_ID

For example:

eth0 08:00:69:13:dc:ec
eth1 08:00:69:13:72:e8

If you have multiple metadata servers in the cluster but only one potential metadata server defined for a given filesystem and that server goes down, the now server-less filesystem goes into a shutdown state. Although the clients maintain membership in the cluster, they will not remount the filesystem automatically when the potential metadata server comes back up. You must manually unmount the filesystem.

If there had been only one potential metadata server in the cluster, the filesystem's clients would have lost membership and gone through a forced shutdown, which automatically unmounts the filesystems.

Forced filesystem shutdown messages do not necessarily imply that xfs_repair should be run. Following is an example of a message that does indicate an XFS file corruption:

XFS read error in file system metadata block 106412416

When a filesystem is forcibly shut down, the log is not empty -- it contains valuable metadata. You must replay it by mounting the filesystem. The log is only empty if the filesystem is unmounted cleanly (that is, not a forced CXFS shutdown, not a crash). You can use the following command line to see an example of the transactions captured in the log file:

xfs_logprint -t device

If you run xfs_repair before mounting the filesystem, xfs_repair will delete all of this valuable metadata.

You should run xfs_ncheck and capture the output to a file before running xfs_repair. If running xfs_repair results in files being placed in the lost+found directory, the saved output from xfs_ncheck may help you to identify the original names of the files.

---

Caution: Always contact SGI technical support before using xfs_repair on CXFS filesystems. See“Repair Filesystems with Care” in Chapter 2.

---

If you think you have a filesystem with real corruption, do the following:

If the GUI will not run, check the following:

This section discusses the following IPMI issues:

BMC Does Not Respond to a ping Command

If the baseboard management controller (BMC) does not respond to a ping(8) command from a remote node, verify that the BMC has a valid IP address assigned. See step 4 in “BMC System Controller” in Appendix D.

---

Note: The BMC will not respond to the ping command when issued from the local node (the node containing the BMC).

---

If the cxfs_admin output appears to be stale (such as after you manually change the port status, in which case the CXFS database is not informed), you can update the CXFS database by running the following command:

# hafence -U

Sending clconfd a SIGTERM signal, the default signal sent by the kill(1) command, will cause the clconfd process to terminate. When the clconfd process terminates on a SIGTERM signal, it is not restarted by cmond and the node will remain in the CXFS cluster membership. All filesystem activity will continue without interruption. However, if clconfd is not running on one or more server-capable administration nodes in the cluster, configuration changes cannot be made in the cluster and CXFS recovery may hang, preventing nodes from joining the cluster membership.

The clconfd process will not run if there is a licensing problem. See “clconfd Daemon Death”.

If you experience problems involving a node being inappropriately taken out of the cluster membership (reset, fenced, or shut down, depending upon how the node fail policy is set), it may be that the CXFS kernel heartbeat timeout setting is inappropriate, especially if your cluster includes a large SGI Altix system (one with more than 64 processors). If you suspect this problem, you should examine the SYSLOG messages to determine the current kernel heartbeat timeout. Membership issues can involve either:

Every kernel heartbeat multicast packet includes a counter value that monotonically increases with every heartbeat sent by the system. You can use a snoop/tcpdump of the private network to look at the counter and determine if the issue is a missing heartbeat or a late heartbeat. A nonconsecutive heartbeat count indicates that the heartbeat was issued by the client-only node but never received by the server-capable administration node. It generally indicates a problem somewhere in the network stack or physical network layer.

Most client-only node heartbeat timeouts are caused by late heartbeats. This is a heartbeat multicast that is not sent out by the client-only node in a timely fashion. It is a strong indicator that the client-only node is suffering from some sort of resource constraint issue. Many common resources are subject to this problem, and detailed system performance analysis is usually required to determine the exact nature of the failure. The following are common problem areas:

To avoid these problems, see “Avoid CXFS Kernel Heartbeat Issues on Large SGI Altix Systems” in Chapter 2.

If file access is slow, it could be due to one of the following problems:

There is only a single active admin login permitted to Fibre Channel switches that are properly configured for CXFS. If there is a defunct admin login remaining, CXFS may be unable to access the switch.

To solve this problem, do the following:

On SGI Altix machines, an NMI system reset will not proceed until there is human intervention. See “System Reset” in Chapter 2.

Rebooting the metadata server without first shutting down CXFS services can cause the metadata server to panic. You must use the proper procedures for shutting down nodes. See “Shut Down Nodes Unobtrusively” in Chapter 2.

The dynamic TCP port assignment used by the fs2d daemon may result in a collision with another service that is already using a fixed port number.

One example that has been observed is Samba hanging during startup when trying to query the CUPS printing system for printer information. Other applications that expect to connect on a well-known port may be affected.

To see which ports fs2d is using, use the following command:

# lsof -i TCP | grep fs2d

If you encounter this problem, restart fs2d with a SIGHUP signal so that it will move to another port:

# /usr/bin/killall -HUP fs2d

You may have to repeat this command until fs2d is assigned to a port that is not in contention.

The Brocade telnet login session can become hung for Brocade switches running firmware version V5* and later. To clear the situation, do the following:

If a CXFS client fails or exits the cluster during the metadata server relocation process, the relocation process and the client recovery are likely to hang. This prevents any clients, including the failed client, from joining the cluster.

Once in this state, it may be possible to resolve the deadlock by resetting or power-cycling the fs2d quorum master. See “Determine the Quorum Master”.

You can expect to see the following messages. They are normal and do not indicate a problem.

If you try to relocate a filesystem and see an error similar to the following cxfs_admin example, it means that relocation has not been enabled:

Error returned from server: feature not enabled (12)
Command "relocate slice1C server=server1" failed during commit: feature not enabled

To allow the relocation to occur, you must enable relocation as specified in “Relocation” in Chapter 1.

If you see an error message such as the following when trying to shut down a node, it means that the node which is the active metadata server for one or more filesystems and was unable to unmount the filesystems, and therefore was unable to shutdown gracefully:

Aug 31 10:06:18 cc-xe kernel: Filesystem "xvm-2": relocation from this host is disabled

Also see “Shutdown Failure”.

If a node is unable to shutdown gracefully, it may be because it was unable to unmount the filesystems for which it is the active metadata server. See “Relocation Error”

If you see messages such as the following on the console or in a message log, it means that the Fibre Channel switch is misconfigured:

controller disable is not supported on loop

CXFS fencing recovery operations do not support loop mode. Verify that all Fibre Channel switches are configured correctly. See the switch documentation for configuration information.

The following messages may be logged by CMS.

CMS excluded cells 0xXXX with incomplete connectivity

Generated when CMS delivers a membership that excluded some new cells that had not established connections with enough cells yet to be admitted. 0xXXX is a binary bitmask of excluded cells.

CMS calculation limited to last membership:configuration change incomplete on cells 0xXXX

Generated when the leader is attempting to make a configuration change current (that is, actually use the change on all nodes), but some cells in the cluster have not yet gotten the configuration change staged (uploaded and ready to be made current). 0xXXX is a binary bitmask of cells that do not yet have the change in their configuration. Changes make their way through the cluster asynchronously, so this situation is expected. It can take a few attempts by the CMS leader before all nodes have the change staged. As long as this situation resolves eventually, there is no problem.

CMS calculation limited to last membership:recovery incomplete

Generated when new members were disallowed due to recovery from the last cell failure that is still being processed.

If the clconfd daemon exits immediately after it starts up, it usually means that the CXFS license key has not been properly installed. Check the end of the clconfd log file (/var/cluster/ha/log/clconfd_nodename) for error messages. For information about licensing error messages, see “License Key Error”.

You must properly install the license keys before you can use CXFS. If you increase the number of CPUs in your system, you may need a new license key. See Chapter 5, “CXFS License Keys”.

The following example system log file message indicates an oversubscribed system:

ALERT: inetd [164] - out of logical swap space during fork while
allocating uarea - see swap(1M)
Availsmem 8207 availrmem 427 rlx freemem 10, real freemem 9

See “Use System Capacity Wisely” in Chapter 2.

Daemons could also be leaking memory in this case. You may need to restart them:

The following message in the system log file indicates a kernel-triggered revocation of CXFS membership:

Membership lost - withdrawing from cluster

You must allow CXFS membership for the local node in this situation. See “Allow Membership of the Local Node with the GUI” in Chapter 10 or “Enable a Node with cxfs_admin” in Chapter 11.

You will see the following error if you try to install CXFS on a server-capable administration node without a valid license key already in place:

Preparing...                ###########################################
[100%]
   1:cxfs_cluster           ###########################################
[100%]
cxfs                      0:off  1:off  2:off  3:on   4:off  5:on   6:off
cluster_cx-exitop: Added CXFS keys to /var/cluster/cdb/cdb.db
cluster_cx-exitop: Added CXFS administration access keys to
/var/cluster/cdb/cdb.db
cxfs license check failed - use '/usr/cluster/bin/cxfslicense -d' for
details

         * * * * * * * * * * I M P O R T A N T * * * * * * * * * * * * *

        CXFS is not properly licensed for this host.  Run
                '/usr/cluster/bin/cxfslicense -d'
        for more detailed license information.
        After fixing the license, please run
                '/bin/true; /etc/init.d/cxfs_cluster restart'.

cluster_cx-exitop: success

If you see the following message in the /var/cluster/ha/log/clconfd_nodename logfile, it means that the CXFS license key was not properly installed:

CXFS not properly licensed for this host.  Run
               '/usr/cluster/bin/cxfslicense -d'
       for detailed failure information. 

If you do not have the CXFS license key properly installed, you will see an error on the console when trying to run CXFS. For example, on a Linux node:

Cluster services:CXFS not properly licensed for this host.  Run
        '/usr/cluster/bin/cxfslicense -d'
for detailed failure information.  After fixing the
license, please run '/etc/init.d/cxfs_cluster restart'.

An error such as the following example will appear in the system log file:

Mar  4 12:58:05 6X:typhoon-q32 crsd[533]: <<CI> N crs 0> Crsd restarted.
Mar  4 12:58:05 6X:typhoon-q32 clconfd[537]: <<CI> N clconf 0> 
Mar  4 12:58:05 5B:typhoon-q32 CLCONFD failed the CXFS license check.Use the 
Mar  4 12:58:05 5B:typhoon-q32    '/usr/cluster/bin/cxfslicense -d'
Mar  4 12:58:05 5B:typhoon-q32 command to diagnose the license problem.

If the clconfd daemon dies right after it starts up, this error may be present.

You must properly install the license key before you can use CXFS. See Chapter 5, “CXFS License Keys”.

If you have conflicting cluster ID numbers at your site, you will see errors such as the following:

WARNING: mtcp ignoring  alive message from 1 with wrong ip addr 128.162.89.34
WARNING: mtcp ignoring  alive message from 0 with wrong ip addr 128.162.89.33

A cluster ID number must be unique. To solve this problem, make the cluster ID numbers unique.

This error can occur if you redefine the cluster configuration and start CXFS services while some nodes have stale information from a previous configuration.

To solve the problem, first try the steps in “Eliminate a Residual Cluster”. If that does not work, reboot the nodes that have stale information. You can determine which nodes have stale information as follows: stale nodes will complain about all of the nodes, but the up-to-date nodes will complain only about the stale nodes. The /var/cluster/ha/log/clconfd_ log file on the stale nodes will also show error messages about SGI_CMS_CONFIG_ID failures.

If there are too many error messages to recognize the stale nodes, reboot every node.

CXFS logs both normal operations and critical errors to the system log file, as well as to individual log files for each log group.

The system log file on server-capable administration nodes is in /var/log/messages.

In general, errors in the system log file file take the following form:

timestamp priority_&_facility : hostname process[ID]: <internal_info> CODE message_text 

For example:

Sep  7 11:12:59 6X:cxfs0 cli[5830]: < E clconf 0> CI_IPCERR_NOSERVER, clconf
ipc: ipcclnt_connect() failed, file /var/cluster/ha/comm/clconfd-ipc_cxfs0

Table 15-2 shows the parts of the preceding message.

The following sections present only the message identifiers and text:

fs2d  - CIS config: multi:IPaddress:5449, delay=1s, incarnation=0x41cc5f0bcdb1c235, flags=

error reading kernel XVM cluster mirror status. Check if XVM module is
started.

CXFS maintains logs for each of the CXFS daemons. For information about customizing these logs, see “Set Log Configuration with the GUI” in Chapter 10.

Log file messages take the following form:

daemon_log timestamp internal_process: message_text

For example:

cad_log:Thu Sep  2 17:25:06.092  cclconf_poll_clconfd: clconf_poll failed with error CI_IPCERR_NOPULSE

Table 15-3, shows the parts in the preceding message.

This section discusses the following:

Thu Jun 3 16:20:45.431 cxfsopus1.example.com cbe_fs2 - cbe_create_node: cannot create new node (RPC error = 9)
  libcdb       - cdb_create_node: error 9 creating child of node 0x60000000000135c0 with subkey "ifd1"

This error means that some nodes have not been created in the cluster database. Error 9 usually means that fs2d is has encountered an internal error while creating that node. To fix the problem, make sure that fs2d is not running on any server-capable administration node and rerun cdbreinit.

Following are common cxfs_admin errors.

Connecting to the local CXFS server...
receiving conflicting bootstrap packets from cluster(s) - cannot identify
server to connect to
gave up trying to connect to server
FATAL: exiting on fatal error

The cxfs_admin command can see multiple clusters. Reconfigure your network so that each cluster's private network subnet is independent of the private network subnet of other clusters. If you have multiple clusters connected to the same public network, use the -i option to identify the cluster name. See “Accessing the Correct Cluster at a Multiple-Cluster Site” in Chapter 11.

Connecting to the CXFS server for the "mycluster" cluster...
Error returned from server: authorization error (8)
Inappropriate privileges to connect to the CXFS server

The host can see the cluster, but does not have permission to connect to it. Use cxfs_admin on another node that does have permission and use the access command to give permission to the first node.

Connecting to the CXFS server for the "mycluster" cluster...
Error returned from server: permissions error (9)
Insufficient privileges to acquire the administration lock

The host only has monitoring privileges and no administration privileges. Use the permission=admin attribute with the access command to grant the host administration rights, or use -r on the cxfs_admin command line.

Connecting to the CXFS server for the "mycluster" cluster...
not receiving bootstrap packets from any cluster - cannot identify server to connect to
gave up trying to connect to server
FATAL: exiting on fatal error

The host is not on the CXFS metadata private network and has not been granted explicit access to the cluster. Grant the host access by using the access command from a server-capable administration node or another host with admin access to the cluster.

The following error indicates that one of the LUNs in this volume is inaccessible. A GPT-labeled LUN in the volume may cause this if GPT labels are not supported on the system:

# /sbin/mount -t cxfs -o 'client_timeout=30s,retry=0,server_list=(server1,server2)' \
/dev/cxvm/stripe93 /mnt/stripe93
 cxfs.util get_subvol_stripe: open(/dev/rcxvm/stripe93) returned -1, errno 19 (Operation not supported by device)
 cxfs.util get_subvol_stripe: Some of the volumes needed for /dev/rcxvm/stripe93 may have a main path that 
   runs through a controller to which this machine is not connected.
 cxfs.util set_xfs_args: get_subvol_stripe failed
 cxfs.util mount_main: set_xfs_args failed

For information about what platforms support GPT labels, see the release notes.

If a node is not allowed to be configured automatically by the cxfs_admin command, errors such as the following will appear in the cxfs_client log:

Aug 06 10:36:51 cxfs_client: cis_cdb_go ERROR: Error returned from server: authorization
error (8)
Aug 06 10:36:51 cxfs_client: cis_autoconf client is not in an autoconf "allowed" policy

For more information about automatic configuration, see “Automatically Configure a Client-Only Node” in Chapter 11.

The cxfs_admin command will also display errors if the node running cxfs_admin does not have administration or monitor access to the cluster. See “Setting cxfs_admin Access Permissions” in Chapter 11.

The following message indicates that the system is unable to open UDP socket connection to another system (in this case, cell ID 4) because the connection attempt is refused:

mesg_connect:cell 4:channel 0:transport 2:connect error -111

Frequently repeated occurrences of this message may indicate an incorrect network configuration or may require reinstalling the CXFS software on either the CXFS client-only node or the server-capable administration node. This message may also occur occasionally when a node is started or shut down.

All server-capable administration nodes in the cluster must run the same version of CXFS except during a rolling upgrade. The following types of errors indicate that the server-capable administration nodes are running different versions of CXFS software:

CXFS cell C (name) mount refused: node is downrev (tag X, has N, need M)
CXFS cell C (name) relocation refused: node is downrev (tag X, has N, need M)

An operation may be forbidden because of a version mismatch between the metadata server and the other node involved. The following operations may fail:

For example, the following message indicates that a client-only node tried to mount a filesystem served by a metadata server that is running an older version of CXFS:

CXFS cell 0 (xo-xe1) mount refused: node is downrev (tag 30, has -1, need 0)

During a rolling upgrade, relocation of a filesystem between server-capable administration nodes running different versions of CXFS may be disabled. Recovery rather than relocation is used to move filesystems to an upgraded server-capable node. See “CXFS Release Versions and Rolling Upgrades” in Chapter 12.

If you see messages like the following on the console or in the system log file, turn off CXFS extents deltas:

EXTENT: a8000002c9e60600 bno 10 not at boundary (got = 9+5)

This is best done on the metadata server by setting the fs.cxfs.cxfs_extents_delta system tunable parameter to 0. A reboot is not required. For more information, see “cxfs_extents_delta” in Appendix E.

If CXFS services to do not restart after a reboot, it may be that the node was marked as INACTIVE in the cluster data base using the Stop CXFS Services function of the GUI, or a disable node:nodename function of cxfs_admin . In this case, issuing a service cxfs_cluster start or service cxfs start will not restart the services.

You must manually start CXFS services. If you use the GUI to restart the services, or enable with cxfs_admin, the configuration will be set so that future reboots will also restart CXFS services.

For information, see “Start CXFS Services with the GUI” in Chapter 10 or “Enable a Node with cxfs_admin” in Chapter 11.

To clear the cluster database on all of the server-capable administration nodes of the cluster, do the following, completing each step on each server-capable administration node before moving to the next step:

---

Caution: This procedure deletes all configuration information.

---

See “Eliminate a Residual Cluster”, to get rid of possible stale cluster configuration in the kernel. If needed, reboot the nodes.

Enter the following individually on every node to reboot the cluster (other than Windows, which uses a different reboot mechanism) :

node# reboot

For information about client-only nodes, see the CXFS 5 Client-Only Guide for SGI InfiniteStorage.

If you want CXFS services to restart whenever the node is rebooted, use the CXFS GUI to start CXFS services or cxfs_admin to enable the node. For information, see “Start CXFS Services with the GUI” in Chapter 10 and “Enable a Node with cxfs_admin” in Chapter 11.

The following are situations that may require a rebooting:

The cxfs_cluster argument to chkconfig controls the other cluster administration daemons and the replicated cluster database:

If turned off, the database daemons will not be started at the next reboot and the local copy of the database will not be updated if you make changes to the cluster configuration on the other nodes. This could cause problems later, especially if a majority of nodes are not running the database daemons.

If the cluster daemons are causing serious trouble and prevent the machine from booting, you can recover the node by booting in single-user mode, turning the argument off and booting in multiuser mode. For example, for an Altix 350:

ELILO boot: linux single
Uncompressing Linux... done
 ...
Skipped services in runlevel S:                   splash
Give root password for login: *****

(none)# /sbin/chkconfig grio2 off (if running GRIOv2)
(none)# /sbin/chkconfig cxfs off
(none)# /sbin/chkconfig cxfs_cluster off
(none)# reboot

For more information, see “chkconfig Arguments” in Chapter 12.

Suppose the following:

To allow node2 to form a cluster by itself, you must do the following:

The cluster will attempt to communicate with the node1 because it is still configured in the cluster, even though it is down. Therefore, it may take some time for the CXFS kernel membership to form and for filesystems to mount.

The commands to stop and restart cluster administration daemons depends upon the platform. See also “Restarting CXFS Services”. For general information about the daemons, see “Kernel Threads” in Appendix A.

To stop and restart cluster administration daemons, enter the following:

---

Note: You could also use the restart option to stop and start.

---

These commands affect the cluster administration daemons only.

---

Caution: When the cluster administration daemons are stopped, the node will not receive database updates and will not update the kernel configuration. This can have very unpleasant side effects. Under most circumstances, the administration daemons should remain running at all times. Use these commands only as directed.

---

See Chapter 13, “Cluster Database Management”. If a node will not join cluster membership, you must reboot it.

To verify general connectivity in a multicast environment, you can execute a ping command on the 224.0.0.1 IP address; 224.0.0.1 is the address for all systems on this subnet multicast.

To verify the CXFS kernel heartbeat, use the 224.0.0.250 IP address. (SGI uses 224.0.0.250 by default; you may need to explicitly set this if you are having problems with name resolution involving cluster_mcast, see below).

---

Note: A node is capable of responding only when the administration daemons (fs2d, cmond, cad, and crsd) or the cxfs_client daemon is running.

---

For example, to see the response for two packets sent from IP address 163.154.17.49 to the multicast address for CXFS kernel heartbeat and ignore loopback, enter the following:

nodeA# ping -c 2 -I 163.154.17.49 -L 224.0.0.250
PING 224.0.0.250 (224.0.0.250): 56 data bytes
64 bytes from 163.154.17.140: icmp_seq=0 ttl=64 time=1.146 ms
64 bytes from 163.154.17.55: icmp_seq=0 DUP! ttl=255 time=1.460 ms
64 bytes from 163.154.17.52: icmp_seq=0 DUP! ttl=255 time=4.607 ms
64 bytes from 163.154.17.50: icmp_seq=0 DUP! ttl=255 time=4.942 ms
64 bytes from 163.154.17.140: icmp_seq=1 ttl=64 time=2.692 ms

----224.0.0.250 PING Statistics----
2 packets transmitted, 2 packets received, +3 duplicates, 0.0% packet
loss
round-trip min/avg/max = 1.146/2.969/4.942 ms

The above output indicates that there is a response from the following addresses:

163.154.17.140
163.154.17.55
163.154.17.52
163.154.17.50

To override the default address, you can use the -c and -m options on client-only nodes; for more information, see the cxfs_client man page.

To override or explicitly set the CXFS multicast address, do the following:

When CXFS is running, you can perform a powercycle on a node with the following cmgr command:

admin powerCycle node nodename

This command uses the CXFS daemons to shut off power to the node and then restart it.

You can perform a powercycle on a node in a cluster even when the CXFS daemons are not running by using the standalone option:

admin powerCycle standalone node nodename

The above command does not go through the crsd daemon.

If the node has not been defined in the cluster database, you can use the following command line (line breaks added here for readability, but it should be all on one line):

admin powerCycle dev_name port|IP_address_or_hostname_of_device of dev_type tty|network|ipmi 
   with sysctrl_type msc|mmsc|l2|l1|bmc

When CXFS is running, you can reset a node with a system controller by using the following cmgr command:

admin reset node hostname  

This command uses the CXFS daemons to reset the specified node.

Even when the CXFS daemons are not running, you can reset a node with a system controller by using the standalone option of the admin reset command:

admin reset standalone node hostname

If you have defined the node but have not defined system controller information for it, you could use the following cmgr commands to connect to the system controller or reset the node:

admin ping dev_name port|IP_address_or_hostname_of_device of dev_type tty|network|ipmi with sysctrl_type msc|mmsc|l2|l1|bmc

admin reset dev_name port|IP_address_or_hostname_of_device of dev_type tty|network|ipmi with sysctrl_type msc|mmsc|l2|l1|bmc

The above command does not go through the crsd daemon.