CXFS supports a client-only node running the Solaris operating system. This chapter contains the following sections:
This section contains the following information about CXFS on Solaris:
In addition to the items listed in “Requirements” in Chapter 1, using a Solaris node to support CXFS requires the following:
Solaris 10 January 06 (patch 118822-25)
Caution: This release of CXFS Solaris only supports the January 06 release (patch 118822-25) of Solaris 10. All other releases of Solaris 10 are not supported and may cause system instabilities when used with CXFS.
The following supported Fibre Channel HBAs (you can use only one vendor for HBA, either LSI Logic or QLogic; you cannot mix HBA vendors):
LSI Logic models using the 01030600 firmware or newer:
LSI7102XP LSI7202XP LSI7402XP LSI7104XP LSI7204XP LSI7404XP QLogic models sold by Sun Microsystems and running with the driver supplied by Sun. (If you have a Qlogic HBA, your system will only access disks with GPT labels. For more information about GPT labels and CXFS, see the CXFS Administration Guide for SGI InfiniteStorage.)
SG-XPCI1FC-QL2 (single-port 2 Gb) SG-XPCI2FC-QF2-Z (dual-port 2 Gb)
Note: CXFS does not automatically detect WWPNs for LSI HBAs or QLogic HBAs. See “I/O Fencing for Solaris”. Any system based on UltraSPARC III, IIIi, or IV with a spare 66-MHz (or faster) PCI slot for a Fibre Channel HBA and a spare 100-Mb/s (or faster) ethernet port for the CXFS private network. CXFS supports a Solaris client only on the SPARC platform. It is not supported on other hardware platforms.
For additional latest information, see the CXFS Solaris release notes.
The following commands are shipped as part of the CXFS Solaris package:
| /usr/cxfs_cluster/bin/cxfs_client |
| /usr/cxfs_cluster/bin/cxfs_info |
| /usr/cxfs_cluster/bin/cxfsdump |
| /usr/sbin/grioadmin |
| /usr/sbin/griomon |
| /usr/sbin/grioqos |
| /usr/cxfs_cluster/bin/xvm |
The cxfs_client and xvm commands are needed to include a client-only node in a CXFS cluster. The cxfs_info command reports the current status of this node in the CXFS cluster.
The pkgadd output lists all software added; see “Solaris Installation Procedure”.
For more information, see the man pages. For additional information about the GRIO commands, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and “GRIO on Solaris”.
The cxfs_client command creates a /var/log/cxfs_client log file. To rotate this log file, use the -z option in the /usr/cxfs_cluster/bin/cxfs_client.options file; see the cxfs_client man page for details.
For information about the log files created on server-capable administration nodes, see the CXFS Administration Guide for SGI InfiniteStorage.
Solaris supports the CXFS mount scripts. See “CXFS Mount Scripts” in Chapter 1 and the CXFS Administration Guide for SGI InfiniteStorage.
Although it is possible to mount a UFS or NFS filesystem on top of a Solaris CXFS filesystem, this is not recommended.
After a crash, attempts to reclaim locks and commit asynchronous writes to a CXFS filesystem from an NFS client may result in a stale file handle.
For optimal performance, you should set the value of the Solaris system tunable parameter maxphys in the /etc/system file. See “maxphys System Tunable for Solaris”.
All disk devices attached to LSI Logic HBAs must be for use only by CXFS disks; do not attach non-disk devices to any Fibre Channel HBA that is configured for CXFS use. This restriction is required because all disk devices on these HBAs (configured for CXFS) make use of the whole disk volume, which must be conveyed to Solaris via modification in the HBA driver to the value returned by the READ_CAPACITY SCSI command.
CXFS does not automatically detect WWPNs for LSI HBAs. See “I/O Fencing for Solaris” for instructions to set up a fencing configuration.
The xvm command displays duplicate entries of physvols. The number of duplicate entries correspond to the devices for each LUN.
See also Appendix B, “Filesystem and Logical Unit Specifications”.
All CXFS files have UNIX mode bits (read, write, and execute) and optionally an access control list (ACL). For more information, see the chmod and setfacl man pages.
If you restore a CXFS file that had an ACL containing only owner-ACL entries (that is, owner/group/other/mask) from a Solaris node, upon restoration one of the following will happen:
When using tar(1), cpio(1), and Legato Networker: The ACL will be lost because these tools behave "intelligently" by not calling acl to set an ACL if the file has only owner/group/other/mask entries. These tools will only set the file mode. However, this does not present a change in functionality because an access permissions check on the mode and the ACL containing only owner entries will give the same result.
When using other backup/restore utilities: A mask will be added to the ACL if the application calls aclfor every file.
A backup/restore utility that calls acl to set an ACL for every file will result in a file being restored with four ACL entries (that is, owner/group/other/mask), even though it may have originally had only three (that is, owner/group/other). This is due to a requirement in getfacl that it receive four ACL entries for the GETACL command to acl. (If fewer than four entries are returned, getfacl will report an error).
| Note: Normally, Solaris filesystem ACLs can have up to 1024 entries for a file and a directory can have 1024 entries as well as an additional 1024 entries for the default ACL. However, CXFS filesystems on Solaris nodes in a multiOS cluster must maintain compatibility with the metadata server. The CXFS filesystems on a Solaris node are limited to a maximum of 25 ACL entries for a file and a maximum total of 50 for a directory (that is, the directory ACL plus the default ACL). |
When using the ls command to display access permissions for a file with an ACL, the mode reported for a CXFS file follows Linux semantics instead of Solaris/UFS semantics.
On Solaris, a UFS file mode reports the group permission as the intersection of the GROUP and MASK entries in the ACL. If the GROUP entry is r-x and the MASK entry is rw-, the group permission will be reported as r--.
The CXFS model calls for reporting the ACL MASK for the group permission in the mode. Therefore, using the example above, the group permission will be reported as rw-. Although it appears that the group has write permission, it does not and an attempt to write to the file will be rejected. You can obtain the real (that is, effective) group permission by using the Solaris getfacl command.
For optimal performance, you should set the value of the Solaris system tunable parameter maxphys in the /etc/system file. Do the following:
Make a backup copy of the /etc/system file.
Note: Exercise extreme caution in changing /etc/system and always make a backup copy. Change the value of maxphys to 0x800000 (hexadecimal) by adding the following to /etc/system :
set maxphys=0x800000
Reboot the Solaris node. This causes the change to take effect.
Verify that the new value for maxphys is in effect by running the following command:
solaris# echo "maxphys/X" | adb -k physmem 1f03f maxphys: maxphys: 800000
The QLogic driver is provided with Solaris 10.
The remainder of this section discusses the following:
These procedures may be performed by you or by a qualified Sun service representative. You must be logged in as root to perform the steps listed in this section.
To install the LSI Logic HBA, perform the following steps. Additional details are provided in the Fibre Channel to PCI-X Host Adapters User's Guide.
Install the LSI Logic HBA into the Solaris system. See the chapter "Installing the Host Adapter" from the Fibre Channel to PCI-X Host Adapters User's Guide.
Bring the system back up.
Install the LSI Logic HBA driver software ( ITImpt, version 5.07.00 or later) according to the instructions in the driver's readme file.
Do the following:
Retrieve the driver package from the following LSI Logic website:
Install the driver package:
solaris# unzip itmpt-5.07.00.zip solaris# uncompress itmpt_install.tar.Z solaris# tar -xvf itmpt_install.tar solaris# cd install solaris# pkgadd -d .
Install the lsi utilities package:
solaris# uncompress lsiutils_v60.tar.Z solaris# tar -xvf lsiutils_v60.tar solaris# cd install solaris# pkgadd -d .
For each target/LUN pair to be used by the LSI Logic HBA, use the lsiprobe utility to add entries to /kernel/drv/ssd.conf.
For example, to add entries for targets 0 through 5 (inclusive), with each of those targets scanning LUNs 0, 2, 4, 5, and 6:
solaris# lsiprobe -a target 0-5 lun 0,2,4-6
Note: If you modify /kernel/drv/ssd.conf, you must reboot the system (as in step 5) in order for changes to take effect. solaris# init 6
After the system has rebooted, verify that the driver attached correctly to the HBA by following the steps “Verifying the HBA Installation”. Do not proceed until the verification succeeds.
After the system reboots, you should verify that the devices were correctly configured by running the Solaris format command. You should see a list of each device you selected.
solaris# format
Searching for disks...done
c2t200400A0B80C268Cd1: configured with capacity of 67.75GB
c2t200400A0B80C268Cd3: configured with capacity of 136.64GB
AVAILABLE DISK SELECTIONS:
0. c0t0d0 /[email protected],600000/[email protected]/[email protected],0
1. c0t1d0 /[email protected],600000/[email protected]/[email protected],0
2. c2t200400A0B80C268Cd1 /[email protected],700000/SUNW,[email protected]/[email protected],0/[email protected],1
3. c2t200400A0B80C268Cd3 /[email protected],700000/SUNW,[email protected]/[email protected],0/[email protected],3
Specify disk (enter its number): |
In this example, disks 2 and 3 are being addressed by the QLogic driver, as indicated by the presence of SUNW,[email protected] in the pathname.
You can also use the luxadm command to view the status of the HBA:
solaris# luxadm -e port /devices/[email protected],700000/SUNW,[email protected]/[email protected],0:devctl CONNECTED /devices/[email protected],700000/SUNW,[email protected],1/[email protected],0:devctl NOT CONNECTED solaris# luxadm probe No Network Array enclosures found in /dev/es Found Fibre Channel device(s): Node WWN:200400a0b80c268b Device Type:Disk device Logical Path:/dev/rdsk/c2t200400A0B80C268Cd1s2 Node WWN:200400a0b80c268b Device Type:Disk device Logical Path:/dev/rdsk/c2t200400A0B80C268Cd3s2 |
The system log and console display may display warning messages similar to the following:
WARNING: /[email protected],700000/SUNW,[email protected]/[email protected],0/[email protected],3 (ssd0): Corrupt label; wrong magic number WARNING: /[email protected],700000/SUNW,[email protected]/[email protected],0/[email protected],1 (ssd1): Corrupt label; wrong magic number |
For QLogic HBA, these messages means that the disk has a bad label or a DVH label, which is not supported. (QLogic HBAs support only GPT labels.)
Similar messages for an LSI Logic HBA will appear on boot for LUNs that have DVH labels. When the XVM module is loaded or started, it installs hooks into the HBA driver and automatically translates the DVH labels into SUN labels (you should not try to relabel the disks with the format command); after XVM translates the labels, you will not see these error messages.
| Note: You can also use the lsiutil command to determine the number of LSI HBAs installed, the model numbers, and firmware versions. |
If you are having trouble with the verification steps, see “New Storage is Not Recognized on Solaris”.
Adding a new RAID can change the fabric name binding. To set up persistent binding, edit the /kernel/drv/itmpt.conf file and add entries of the following format, one for each target, where portWWN is the port world wide name of the RAID controller:
target-X-wwn="portWWN" |
For example:
target-0-wwn="200800a0b8184c8e" target-1-wwn="200900a0b8184c8d" target-2-wwn="200400a0b80c268c" target-3-wwn="200500a0b80c268c" |
In this example, target-0 will be bound to the device with the port WWN 200800a0b8184c8e and the resulting devices will be:
/[email protected],700000/IntraServer,[email protected]/[email protected],* |
This section provides an overview of the steps that you or a qualified Sun service representative will perform on your Solaris nodes prior to installing the CXFS software. It contains the following sections:
The following procedure provides an overview of the steps required to add a private network to the Solaris system. A private network is required for use with CXFS. See “Use a Private Network” in Chapter 2.
You may skip some steps, depending upon the starting conditions at your site. For details about any of these steps, see the Solaris documentation.
If your system is already operational and on the network, skip to step 2.
If your Solaris system has never been set up, bring the system to single-user mode. For example, go to the PROM prompt and boot the Solaris node into single-user mode:
> boot -s
As a last resort, you can reach the PROM prompt by pressing the L1-A (or Stop-A) key sequence.
Edit the /etc/inet/ipnodes file so that it contains entries for each node in the cluster and its private interfaces.
The /etc/inet/ipnodes file has the following format, where primary_hostname can be the simple hostname or the fully qualified domain name:
IP_address primary_hostname aliases
You should be consistent when using fully qualified domain names in the /etc/inet/ipnodes file. If you use fully qualified domain names on a particular node, then all of the nodes in the cluster should use the fully qualified name of that node when defining the IP/hostname information for that node in their /etc/inet/ipnodes file.
The decision to use fully qualified domain names is usually a matter of how the clients (such as NFS) are going to resolve names for their client server programs, how their default resolution is done, and so on.
Even if you are using the domain name service (DNS) or the network information service (NIS), you must add every IP address and hostname for the nodes to /etc/inet/ipnodes on all nodes. For example:
190.0.2.1 server1.company.com server1 190.0.2.3 stocks 190.0.3.1 priv-server1 190.0.2.2 server2.company.com server2 190.0.2.4 bonds 190.0.3.2 priv-server2
You should then add all of these IP addresses to /etc/inet/ipnodes on the other nodes in the cluster.
For more information, see the hosts, named, and nis man pages.
Note: Exclusive use of NIS or DNS for IP address lookup for the nodes will reduce availability in situations where the NIS or DNS service becomes unreliable. For more information, see “Understand Hostname Resolution and Network Configuration Rules” in Chapter 2.
Edit the /etc/nsswitch.conf file so that local files are accessed before either NIS or DNS. That is, the ipnodes line in /etc/nsswitch.conf must list files first.
ipnodes: files nis dns
(The order of nis and dns is not significant to CXFS, but files must be first.)
Determine the name of the private interface by using the ifconfig command as follows:
solaris# ifconfig -a
If the second network does not appear, it may be that a network interface card must be installed in order to provide a second network, or it may be that the network is not yet initialized.
For example, on an Ultra Enterprise 250, the integrated Ethernet is hme0; this is the public network. The following ifconfig output shows that only the public interface exists:
solaris# ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255 ether 8:0:20:d2:29:c5
If the second network does not appear, do the following:
If you do not have the PCI card installed, install it. Refer to your PCI documentation for instructions.
If your card is already installed, skip to step b.
Use the output from the dmesg command to determine the interface name for the private network; look for the network interface that immediately follows the public network; you may wish to search for Found. For example:
solaris# dmesg Feb 6 09:38:36 ue250 last message repeated 42 times Feb 6 11:38:40 ue250 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0 Feb 6 11:38:40 ue250 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/[email protected] Feb 6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme0 : PCI IO 2.0 (Rev Id = c1) Found Feb 6 11:38:41 ue250 genunix: [ID 936769 kern.info] hme0 is /[email protected],4000/[email protected],1 Feb 6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme1 : PCI IO 2.0 (Rev Id = c1) Found Feb 6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme1 : Local Ethernet address = 8:0:20:cc:43:48 Feb 6 11:38:41 ue250 pcipsy: [ID 370704 kern.info] PCI-device: SUNW,[email protected],1, hme1 Feb 6 11:38:41 ue250 genunix: [ID 936769 kern.info] hme1 is /[email protected],2000/SUNW,[email protected],1
The second network is hme1; this is the private network, and is displayed after hme0 in the dmesg output. In this example, hme1 is the value needed in step c and in step 5 below.
Initialize the private network's interface by using the ifconfig command as follows, where interface is the value determined in step b:
ifconfig interface plumb
For example:
solaris# ifconfig hme1 plumb
After performing the plumb, the hme1 interface will appear in the ifconfig output, although it will not contain the appropriate information (the correct information will be discovered after the system is rebooted later in step 8). For example, at this stage you would see the following:
solaris# ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255 ether 8:0:20:d2:29:c5 hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 0.0.0.0 netmask ff000000 broadcast 255.0.0.0 ether 8:0:20:d2:29:c5
Create a file named /etc/hostname.interface, where interface is the value determined in step 4. This file must contain the name of the private network. For example:
solaris# cat /etc/hostname.hme1 cxfssun3-priv
Note: In this scenario, /etc/hostname.hme0 must contain the same value as the /etc/nodename file. For example: solaris# cat /etc/hostname.hme0 cxfssun3 solaris# cat /etc/nodename cxfssun3
Edit the /etc/netmasks file to include the appropriate entries.
(Optional) Edit the /.rhosts file if you want to use remote access or if you want to use the connectivity diagnostics provided with CXFS. Ensure that the mode of the .rhosts file is set to 600 (read and write access for the owner only).
Make sure that the /.rhosts file on each Solaris node allows all of the nodes in the cluster to have access to each other. The connectivity tests execute a ping command from the local node to all nodes and from all nodes to the local node. To execute ping on a remote node, CXFS uses rsh as user root.
For example, suppose you have a cluster with three nodes: admin0, solaris1, and solaris2. The /.rhosts files could be as follows (the prompt denotes the node name):
admin0# cat /.rhosts solaris1 root solaris1-priv root solaris2 root solaris2-priv root solaris1# cat /.rhosts admin0 root admin0-priv root solaris2 root solaris2-priv root solaris2# cat /.rhosts admin0 root admin0-priv root solaris1 root solaris1-priv root
solaris# init 6
At this point, ifconfig will show the correct information for the private network.
For example:
ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255 ether 8:0:20:d2:29:c5 hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3 inet 10.1.1.36 netmask ffffff00 broadcast 10.1.1.255 ether 8:0:20:d2:29:c5
For each private network on each Solaris node in the pool, verify access with the Solaris ping command. Enter the following, where nodeIPaddress is the IP address of the node:
solaris# /usr/sbin/ping -s -c 3 nodeIPaddress |
solaris# /usr/sbin/ping -s -c 3 128.162.2.91 PING 128.162.2.91: 56 data bytes 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=0. time=0. ms 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=1. time=0. ms 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=2. time=0. ms 64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=3. time=0. ms |
Also execute a ping on the public networks. If ping fails, follow these steps:
Verify that the network interface was configured up using ifconfig ; for example:
solaris# /usr/sbin/ifconfig eri0 eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2 inet 128.162.2.127 netmask ffffff00 broadcast 128.162.2.255 ether 0:3:ba:d:ad:77
In the first output line above, UP indicates that the interface was configured up.
Verify that the cables are correctly seated.
Repeat this procedure on each node.
The CXFS software will be initially installed and configured by SGI personnel. This section provides an overview of those procedures. You can use the information in this section to verify the installation.
Installing the CXFS client software for Solaris requires approximately 20 MB of space.
To install the required software on a Solaris node, SGI personnel will do the following:
Read the SGI InfiniteStorage Software Platform release notes, CXFS general release notes, and CXFS Solaris release notes in the /docs directory on the ISSP DVD and any late-breaking caveats on Supportfolio
Verify that the node has been upgraded to Solaris 10 (also known as SunOS 5.10) according to the Solaris installation guide. Use the following command to display the currently installed system:
solaris# uname -r
This command should return a value of 5.10.
Transfer the client software that was downloaded onto a server-capable administration node during its installation procedure using ftp, rcp, or scp . The location of the package on the server will be as follows:
/usr/cluster/client-dist/CXFS_VERSION/solaris/10/noarch/SGIcxfs-sol10-vCXFS_VERSION.pkg
Install the package:
solaris# pkgadd -d SGIcxfs*.pkg
You must select the package to be installed and then confirm that you want allow the installation of the packages with superuser permission. For example:
solaris# pkgadd -d SGIcxfs*.pkg The following packages are available: 1 SGIcxfs SGI CXFS client software (sparc) 5.0.0.3 Select package(s) you wish to process (or 'all' to process all packages). (default: all) [?,??,q]: 1 Processing package instance <SGIcxfs> from </tmp/SGIcxfs-sol10-v5.0.0.3.pkg> SGI CXFS client software(sparc) 5.0.0.3 /* * Copyright (c) 2008 Silicon Graphics, Inc. ALL RIGHTS RESERVED ... */ Using </> as the package base directory. ## Processing package information. ## Processing system information. 3 package pathnames are already properly installed. ## Verifying disk space requirements. ## Checking for conflicts with packages already installed. ## Checking for setuid/setgid programs. This package contains scripts which will be executed with super-user permission during the process of installing this package. Do you want to continue with the installation of <SGIcxfs> [y,n,?] y Installing SGI CXFS client software as <SGIcxfs> ## Installing part 1 of 1. /etc/init.d/cxfs_cluster /etc/rc0.d/K28cxfs_cluster <symbolic link> /etc/rc1.d/K28cxfs_cluster <symbolic link> /etc/rc2.d/S77cxfs_cluster <symbolic link> /etc/rc3.d/S77cxfs_cluster <symbolic link> /etc/rcS.d/K28cxfs_cluster <symbolic link> /usr/cxfs_cluster/bin/cxfs_client /usr/cxfs_cluster/bin/cxfs_info /usr/cxfs_cluster/bin/cxfsdump /usr/cxfs_cluster/bin/cxfslicense /usr/cxfs_cluster/bin/frametest /usr/cxfs_cluster/bin/xvm /usr/kernel/drv/cell.conf /usr/kernel/drv/sparcv9/cell /usr/kernel/drv/sparcv9/xvm /usr/kernel/drv/xvm.conf /usr/kernel/fs/sparcv9/cxfs /usr/kernel/misc/sparcv9/hba_hook /usr/lib/sparcv9/libgrio.so /usr/sbin/clmount /usr/sbin/grioadmin /usr/sbin/griomon /usr/sbin/grioqos /usr/sbin/idbg /usr/share/man/man1/xvm.1 /usr/share/man/man1m/cxfs_client.1m /usr/share/man/man1m/cxfs_info.1m /usr/share/man/man1m/frametest.1m /var/cluster/cxfs_client-scripts/cxfs-post-mount /var/cluster/cxfs_client-scripts/cxfs-post-umount /var/cluster/cxfs_client-scripts/cxfs-pre-mount /var/cluster/cxfs_client-scripts/cxfs-pre-umount /var/cluster/cxfs_client-scripts/cxfs-reprobe [ verifying class <none> ] ## Executing postinstall script. Starting CXFS services... cxfs_client daemon started
To verify that the CXFS software has been installed properly, use the pkginfo command as follows:
pkginfo -l SGIcxfs |
For example, the following output indicates that the CXFS package installed properly:
solaris# pkginfo -l SGIcxfs
PKGINST: SGIcxfs
NAME: SGI CXFS client software
CATEGORY: system
ARCH: sparc
VERSION: 5.0.0.3
BASEDIR: /
VENDOR: Silicon Graphics Inc.
PSTAMP: cxfssun120080213222059
INSTDATE: Feb 14 2008 07:15
STATUS: completely installed
FILES: 38 installed pathnames
5 directories
22 executables
20179 blocks used (approx) |
I/O fencing is required on Solaris nodes in order to protect data integrity of the filesystems in the cluster.
The /etc/fencing.conf file enumerates the WWPN for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for the HBA WWPN as a 64-bit hexadecimal number.
| Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric. |
If used, /etc/fencing.conf must contain a simple list of WWPNs, one per line. You must update it whenever the HBA configuration changes, including the replacement of an HBA.
Do the following:
Set up the switch and HBA. See the release notes for supported hardware.
If the HBA is a Sun QLogic card, use the /usr/sbin/fcinfo command. For example:
# /usr/sbin/fcinfo hba-port | grep 'HBA Port WWN:' | cut -d':' -f2 210000e08b86d53c 210100e08ba6d53c
For more information, see the QLogic documentation.
If the HBA is an LSI card, you can use the lsiutil command to scan for devices. For example for ports 1 and 2:
# lsiutil -p 1 8 | grep 'FC949X Port' | awk '{print $3}' 100000062b114f50 # lsiutil -p 2 8 | grep 'FC949X Port' | awk '{print $3}' 100000062b114f51
For more information, see the LSI documentation.
If either of the above do not work, do the following:
Follow the Fibre Channel cable on the back of the node to determine the port to which it is connected in the switch. Ports are numbered beginning with 0. (For example, if there are 8 ports, they will be numbered 0 through 7.)
Connect to the switch via telnet and log in as user admin. (On Brocade switches, the password is password by default).
Use switchshow command. For example:
brocade04:admin> switchshow switchName: brocade04 switchType: 2.4 switchState: Online switchRole: Principal switchDomain: 6 switchId: fffc06 switchWwn: 10:00:00:60:69:12:11:9e switchBeacon: OFF port 0: sw Online F-Port 20:00:00:01:73:00:2c:0b port 1: cu Online F-Port 21:00:00:e0:8b:02:36:49 port 2: cu Online F-Port 21:00:00:e0:8b:02:12:49 port 3: sw Online F-Port 20:00:00:01:73:00:2d:3e port 4: cu Online F-Port 21:00:00:e0:8b:02:18:96 port 5: cu Online F-Port 21:00:00:e0:8b:00:90:8e port 6: sw Online F-Port 20:00:00:01:73:00:3b:5f port 7: sw Online F-Port 20:00:00:01:73:00:33:76 port 8: sw Online F-Port 21:00:00:e0:8b:01:d2:57 port 9: sw Online F-Port 21:00:00:e0:8b:01:0c:57 port 10: sw Online F-Port 20:08:00:a0:b8:0c:13:c9 port 11: sw Online F-Port 20:0a:00:a0:b8:0c:04:5a port 12: sw Online F-Port 20:0c:00:a0:b8:0c:24:76 port 13: sw Online L-Port 1 public port 14: sw No_Light port 15: cu Online F-Port 21:00:00:e0:8b:00:42:d8
The WWPN is the hexadecimal string to the right of the port number. For example, the WWPN for port 0 is 2000000173002c0b (you must remove the colons from the WWPN reported in the switchshow output to produce the string to be used in the fencing file).
Edit or create /etc/fencing.conf and add the WWPN for the port determined in step 2. (Comment lines begin with #.)
For dual-ported HBAs, you must include the WWPNs of any ports that are used to access cluster disks. This may result in multiple WWPNs per HBA in the file; the numbers will probably differ by a single digit.
For example, if you determined that port 0 is the port connected to the switch, your fencing file should contain the following:
# WWPN of the HBA installed on this system # 2000000173002c0b
To configure fencing, see the CXFS Administration Guide for SGI InfiniteStorage.
The /etc/init.d/cxfs_cluster script will be invoked automatically during normal system startup and shutdown procedures. This script starts and stops the cxfs_client daemon.
To start cxfs_client manually, enter the following:
solaris# /etc/init.d/cxfs_cluster start |
To stop cxfs_client manually, enter the following:
solaris# /etc/init.d/cxfs_cluster stop |
To stop and then start cxfs_client manually, enter the following:
solaris# /etc/init.d/cxfs_cluster restart |
This section contains the following:
| Note: Before upgrading CXFS software, ensure that no applications on the node are accessing files on a CXFS filesystem. |
To upgrade CXFS on a Solaris system, do the following:
Remove the current package:
solaris# pkgrm SGIcxfs The following package is currently installed: SGIcxfs SGI CXFS client software (sparc) releaselevel Do you want to remove this package? [y,n,?,q] y # Removing installed package instance <SGIcxfs> This package contains scripts which will be executed with super-user permission during the process of removing this package. Do you want to continue with the removal of this package [y,n,?,q] y # Verifying package dependencies ...
Reboot the Solaris system:
solaris# reboot
Obtain the CXFS update software from Supportfolio according to the directions in CXFS Administration Guide for SGI InfiniteStorage
Follow the installation instructions to install the new package. See “Client Software Installation for Solaris”.
You can modify the behavior of the CXFS client daemon ( cxfs_client) by placing options in the /usr/cxfs_cluster/bin/cxfs_client.options file. The available options are documented in the cxfs_client man page.
| Caution: Some of the options are intended to be used internally by SGI only for testing purposes and do not represent supported configurations. Consult your SGI service representative before making any changes. |
To see if cxfs_client is using the options in cxfs_client.options, enter the following:
solaris# ps -ef | grep cxfs |
If you make changes to your storage configuration, you must rerun the HBA utilities to reprobe the storage. See “HBA Installation for Solaris”.
CXFS supports guaranteed-rate I/O (GRIO) version 2 on the Solaris platform. Application bandwidth reservations must be explicitly released by the application before exit. If the application terminates unexpectedly or is killed, its bandwidth reservations are not automatically released and will cause a bandwidth leak. If this happens, the lost bandwidth could be recovered by rebooting the client node.
For more information, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and the Guaranteed-Rate I/O Version 2 Guide.
Following is an example of the /etc/failover2.conf file on Solaris using a QLogic HBA:
/[email protected],700000/SUNW,[email protected]/[email protected],0/[email protected],1 affinity=1 preferred /[email protected],700000/SUNW,[email protected],1/[email protected],0/[email protected],1 affinity=1 /[email protected],700000/SUNW,[email protected]/[email protected],0/[email protected],1 affinity=2 /[email protected],700000/SUNW,[email protected],1/[email protected],0/[email protected],1 affinity=2 |
In this case:
SUNW,[email protected] is the first port on the PCI card
SUNW,[email protected],1 is the second port on the PCI card
200400a0b80c268c is controller A on the TP9XXX
200500a0b80c268c is controller B on the TP9XXX
Following is an example using an LSI HBA:
<XVM physvol phys/cc_is4000-lun0>
[email protected],2000/IntraServer,[email protected],1/[email protected],0 <dev 130> affinity=1
[email protected],2000/IntraServer,[email protected],1/[email protected],0 <dev 146> affinity=0 preferred
<XVM physvol phys/cc_is4000-lun1>
[email protected],2000/IntraServer,[email protected],1/[email protected],1 <dev 738> affinity=1 preferred
[email protected],2000/IntraServer,[email protected],1/[email protected],1 <dev 930> affinity=0
<XVM physvol phys/cc_is4000-lun2>
[email protected],2000/IntraServer,[email protected],1/[email protected],2 <dev 746> affinity=1
[email protected],2000/IntraServer,[email protected],1/[email protected],2 <dev 938> affinity=0 preferred
<XVM physvol phys/cc_is4000-lun3>
[email protected],2000/IntraServer,[email protected],1/[email protected],3 <dev 754> affinity=1 preferred
p[email protected],2000/IntraServer,[email protected],1/[email protected],3 <dev 946> affinity=0 |
For more information, see “XVM Failover and CXFS” in Chapter 1, the comments in the /etc/failover2.conf file, CXFS Administration Guide for SGI InfiniteStorage, and the XVM Volume Manager Administrator's Guide.
This section contains the following:
For general troubleshooting information, see Chapter 11, “General Troubleshooting” and Appendix D, “Error Messages”.
Confirm that the cxfs_client is not running. The following command would list the cxfs_client process if it were running:
solaris# ps -ef | grep cxfs_client |
Check the cxfs_client log file for errors.
Restart cxfs_client as described in “Start/Stop cxfs_client for Linux” in Chapter 5 and watch the cxfs_client log file for errors.
If cxfs_info reports that cms is up but XVM or the filesystem is in another state, then one or more mounts is still in the process of mounting or has failed to mount.
The CXFS node might not mount filesystems for the following reasons:
The client may not be able to see all the LUNs. This is usually caused by misconfiguration of the HBA or the SAN fabric:
Can the HBA see all of the LUNs for the filesystems it is mounting?
Can the operating system kernel see all of the LUN devices?
The cxfs_client daemon may not be running. See “The cxfs_client Daemon is Not Started on Solaris ”.
The filesystem may have an unsupported mount option. Check the cxfs_client.log for mount option errors or any other errors that are reported when attempting to mount the filesystem.
The cluster membership (cms), XVM, or the filesystems may not be up on the node. Execute the /usr/cxfs_cluster/bin/cxfs_info command to determine the current state of cms, XVM, and the filesystems. If the node is not up for each of these, then check the /var/log/cxfs_client log to see what actions have failed.
Do the following:
If cms is not up, check the following:
Is the node is configured on the server-capable administration node with the correct hostname?
Has the node been added to the cluster and enabled? See “Verifying the Cluster Status” in Chapter 10.
If XVM is not up, check that the HBA is active and can see the LUNs.
If the filesystem is not up, check that one or more filesystems are configured to be mounted on this node and check the /var/log/cxfs_client file for mount errors.
If you have a problem with an HBA, verify that you enabled fabric mode. See “Recognizing Storage Changes for Solaris”.
The /var/log/cxfs_client log file may become quite large over a period of time if the verbosity level is increased. To manually rotate this log file, use the -z option in the /usr/cxfs_cluster/bin/cxfs_client.options file.
For information about the log files created on server-capable administration nodes, see the CXFS Administration Guide for SGI InfiniteStorage.
To view the CXFS heartbeat value on Solaris, use the following:
# echo mtcp_hb_period/D | adb -k physmem 3df86 mtcp_hb_period: mtcp_hb_period: 600 |
Using the -k option to the adb(1) debugger causes it to attach to a live kernel. Echoing the command allows you to put it on a single line.
For example, to reset the value to 15 seconds, enter the following (the value is in Hz):
# echo mtcp_hb_period/W0t1500 | adb -kw physmem 3df86 mtcp_hb_period: 0x258 = 0x5dc |
When reporting a problem about a CXFS Solaris node to SGI, you should retain the following information:
If there is a system panic, retain the system core file in /var/crash/hostname on a Solaris node.
A list of the installed CXFS packages. Use the pkginfo command as follows:
# pkginfo -l SGIcxfs
A list of the Solaris patches that have been installed. Use the showrev command. The showrev command without options prints a summary and the -p option lists the revision information about patches.
A list of the loaded Solaris kernel modules and versions. Use the modinfo command.
Output about the cluster obtained from the cxfsdump utility run on a server-capable administration node. When run in local mode on a Solaris node, it stores information in /var/cluster/cxfsdump-data/ nodename.tar.gz.
Output from the LSI /usr/sbin/lsiutil command, which displays the number of LSI HBAs installed, the model numbers, and firmware versions.
If any of the above Solaris tools are not currently installed on your Solaris system, you should install them.