Use df to make sure that all filesystems on shared disks are mounted only on one node; no filesystem should be simultaneously mounted by two nodes.

Look in /var/adm/SYSLOG or the file you specified in /etc/syslog.conf (see the section “Messages From IRIS FailSafe” in Chapter 6) on both nodes for causes of failure.

Diagnose and repair the problem using the information in the remainder of this appendix.

If the failure caused a failover and the failed node is in standby state, after repairing the problem you can bring both nodes back to normal state by following the procedure in the section “Moving a Node From Standby State to Normal or Degraded State” in Chapter 6.

Make sure that IRIS FailSafe is chkconfig'd on.

---

Note: If two failovers have occurred within a period of time specified by MIN_UPTIME in /etc/init.d/failsafe (300 seconds, by default) IRIS FailSafe software automatically performs chkconfig failsafe off.

---

Make sure that /var/ha/ha.conf is identical on both nodes in the cluster by entering this command on each node:

# /usr/etc/ha_cfgchksum
0x12a2390e

The checksums output by the commands should be identical.

Check the format and contents of /var/ha/ha.conf using the ha_cfgverify command:

# /usr/etc/ha_cfgverify 

Ensure that there are no errors from ha_cfgverify.

Verify the network interfaces and serial connections using the procedures in the sections “Testing the Public Network Interfaces” and “Testing the Serial Connections” in Chapter 5.

Look at /var/adm/SYSLOG to see what errors are printed out by the IRIS FailSafe daemons. When a node in the IRIS FailSafe cluster starts up normally, the following SYSLOG messages appear:

ha_appmon[6141]: Received XRELEASE_PEER
ha_nc[6135]: Received JOINING
ha_nc[6135]: New state: NC_JOINING
ha_nc[6135]: Received REJOIN
ha_appmon[6141]: Received XACQUIRE
ha_appmon[6141]: Received START_REMMON
ha_nc[6135]: New state: NC_NORMAL

Verify that the SCSI host IDs of the two nodes are different (by convention, 0 and 2) by running nvram.

To change the SCSI host ID, enter this command:

# nvram -v scsihostid id 

Verify that the SCSI IDs of all disks and other peripherals on the same SCSI bus have distinct SCSI unit numbers and that they are different from the SCSI host IDs of the two nodes in the cluster (usually 0 and 2).

---

Note: This convention works when only one internal disk per node is used. If a second internal disk is used in either node, that disk's SCSI ID cannot be identical to either SCSI host ID.

---

Get information about the interface from this ifconfig command:

# /usr/etc/ifconfig ec0
ec0: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,FILTMULTI,MULTICAST>
        inet 190.0.2.2 netmask 0xffffff00 broadcast 190.0.2.255

The UP in the first line of output indicates that the interface is configured up.

If the interface is not configured up, add the interface to the /etc/config/netif.options file as described in the section “Configuring Network Interfaces” in Chapter 3.

If the interface is configured up, check if the network interface is working by entering this command:

# /usr/etc/netstat -I ec0 

Check the output to see if there are input errors or output errors for the interface.

To stop the automatic communication on the serial line, enter this command:

# /usr/etc/ha_admin -m stop hostname 
ha_admin: Stopped monitoring the serial connection to hostname

Manually send messages on the serial line by entering this command:

# /usr/etc/ha_spng -i 10 -f reset-tty -d sys-ctlr-type -w password 

reset-tty is the value of the reset-tty parameter for this node in the configuration file /var/ha/ha.conf. sys-ctlr-type is the value of the sys-ctlr-type parameter for this node in /var/ha/ha.conf. The -d sys-ctlr-type option is omitted if there is no sys-ctlr-type parameter or it is set to CHAL. password is the unencrypted password for this node's system controller. The -w password option is omitted if the node is a CHALLENGE node or if it is an Origin node with the default system controller password.

Check the return code from the command in step 2:

# echo variable 
0

If you are using csh, variable is $status. If you are running sh, variable is $?. The zero output indicates normal operation.

If the return code was zero, restart automatic communication on the serial line by entering this command:

# /usr/etc/ha_admin -m start hostname 
ha_admin: Started monitoring the serial connection to hostname

There is no problem with the serial connection, so skip the remainder of this procedure.

If the return code from ha_spng was non-zero, try re-seating the serial cable connectors.

Re-test the serial line by performing steps 2 and 3.

If re-seating the cables didn't work, replace the cable by following the directions in the section “Replacing the Serial Cable” in Chapter 8.

Verify that the node is licensed for plexing and that the plexing software is installed:

# xlv_mgr
xlv_mgr> show config
Allocated subvol locks: 30 locks in use: 7
Plexing license: present
Plexing support: present
Maximum subvol block number: 0x7fffffff

If you have just installed the plexing software, you must reboot the system for the plexing support to be included in the kernel.

Verify that the node sees the volume. In an IRIS FailSafe environment, a node accesses (and mounts filesystems on) only those XLV volumes that it owns. To see all the volumes in the cluster, enter these commands:

xlv_mgr> show all
Volume: vol1 (complete)
Volume: vol2 (complete)
Volume: shared_vol (complete)
xlv_mgr> quit

This command shows all the XLV volumes in the cluster.

If the cluster is using a CHALLENGE RAID storage system, stop the RAID agent by entering this command:

# /etc/init.d/raid5 stop 

To see volumes owned by a node, enter this command on that node:

# xlv_assemble -ln

VOL vol2        flags=0x1, [complete]
DATA    flags=0x0()     open_flag=0x0() device=(192, 4)
PLEX 0  flags=0x0
VE 0    [active]
        start=0, end=687999, (cat)grp_size=1
        /dev/dsk/dks5d9s0 (688000 blks)
PLEX 1  flags=0x0
VE 0    [active]
        start=0, end=687999, (cat)grp_size=1
        /dev/dsk/dks5d9s1 (688000 blks)

This command displays only the volumes owned by this node.

If a volume is owned by the wrong node, change the ownership of a volume (for example, make vol2 owned by xfs-ha2) using xlv_mgr after first unmounting the filesystem mounted on that volume (vol2):

# umount /vol2
# xlv_mgr
xlv_mgr> change nodename xfs-ha2 vol2
set node name "xfs-ha2" for object "vol2" done

Run xlv_assemble -l on both nodes:

# xlv_assemble -l 

Restart the RAID agent, if you stopped it in step 3, by entering this command:

# /etc/init.d/raid5 start 

In your /var/ha/ha.conf file, look for the filesystem block for a filesystem that does not mount successfully. For example:

filesystem fs1
{
        mount-point = /shared
        mount-info
        {
                fs-type = xfs
                volume-name = vol1
                mode = rw,noauto
        }
}

This filesystem block is for the filesystem mounted at /shared.

Look for the volume block for this filesystem; it is the volume block whose label is vol1, the value of volume-name. For example:

volume vol1
{
        server-node = xfs-ha1
        backup-node = xfs-ha2
        devname = vol1
        devname-owner = root
        devname-group = sys
        devname-mode = 0600
}

Follow the procedure in the section “Trouble Accessing a Network Interface” in this appendix to make sure that the volume is owned by this node.

Mount the filesystem with this command:

# mount -txfs -rw,noauto /dev/xlv/vol1 /shared 

Unmount the filesystem with this command:

# umount /shared

---

Note: Do not omit this step. Data corruption could result.

---

Repeat steps 1 through 5 for every filesystem that doesn't mount successfully.

On the other node, repeat steps 1 through 6. While following the procedure in the section “Trouble Accessing a Network Interface,” do change the owner of the volume to the second node.

Verify that the value of ip-address is a high-availability IP address. See the section “NFS Blocks” in Chapter 4 for more information.

Verify that /shared1 is mounted. If it is not, follow instructions in the section “Trouble Accessing a Network Interface” in this appendix.

Export the filesystem by entering this command:

# exportfs -i -o rw /shared1 

Verify that the filesystem is exported:

# exportfs
/shared1 -rw 

From a client, mount the filesystem and verify that you can access it. For example

# mount -tnfs -rw 190.0.2.3:/shared1 /shared1

On the client, change to /shared1, verify that the filesystem has been mounted, then unmount it:

# cd /shared1
# ls
# cd /
# umount /shared1

Unexport and unmount the filesystem:

# exportfs -u /shared1
# umount /shared1

---

Note: Do not omit this step. Data corruption could result.

---

Repeat steps 2 through 7 with the filesystem mounted on the other node.

If you have multiple Netscape servers, verify that the file /etc/config/ns_fasttrack.options file exists (it is created by IRIS FailSafe when it starts Netscape servers):

# ls /etc/config/ns_fasttrack.options

Enable and start the Netscape FastTrack server if used:

# chkconfig ns_fasttrack on
# /etc/init.d/ns_fasttrack start 

Enable and start the Netscape Enterprise server if used:

# chkconfig ns_enterprise on
# /etc/init.d/ns_enterprise start 

Run a Web browser, such as Netscape, and try to access some Web pages exported by the server.

Shut the cluster down by following the procedure in the section “Shutting Down IRIS FailSafe” in Chapter 6.

Re-enter the command with the same command options that failed. In the example above, the command is the call to /sbin/xlv_mgr.

Rerun the appropriate script from /var/ha/actions (in this case, /var/ha/actions/takeback). For example:

# /var/ha/actions/takeback `/usr/etc/ha_cfgchksum`

Make the appropriate fixes, which are most likely to be fixing errors in configuration, and bring the cluster back up.

Check to see what options the network status monitor daemon, /usr/etc/rpc.statd, was started with by entering this command:

# ps -ef | grep rpc.statd
    root   211     1  0   May 31 ?       0:00 /usr/etc/rpc.statd -h
    root  4215  4213  2 14:48:57 ttyq3   0:00 grep rpc.statd 

Look for the -h option (shown in this output). The error message shown above appears if the -h option wasn't used.

If the -h option wasn't used, add -h to the first line of the file /etc/config/statd.options as described in the section “Configuring NFS Filesystems” in Chapter 3.

If you edited /etc/config/statd.options, restart rpc.statd by entering this command:

# /etc/init.d/network start