Chapter 8. Solaris Platform

CXFS supports a client-only node running the Solaris operating system. This chapter contains the following sections:

CXFS on Solaris

This section contains the following information about CXFS on Solaris:

Requirements for Solaris

In addition to the items listed in “Requirements” in Chapter 1, using a Solaris node to support CXFS requires the following:

  • Solaris operating system:

    • Solaris 10 January 06 (patch 118822-25)


      Caution: This release of CXFS Solaris only supports the January 06 release (patch 118822-25) of Solaris 10. All other releases of Solaris 10 are not supported and may cause system instabilities when used with CXFS.


  • The following supported Fibre Channel HBAs (you can use only one vendor for HBA, either LSI Logic or QLogic; you cannot mix HBA vendors):

    • LSI Logic models using the 01030600 firmware or newer:

      LSI7102XP
      LSI7202XP
      LSI7402XP
      LSI7104XP
      LSI7204XP
      LSI7404XP

    • QLogic models sold by Sun Microsystems and running with the driver supplied by Sun. (If you have a Qlogic HBA, your system will only access disks with GPT labels. For more information about GPT labels and CXFS, see the CXFS Administration Guide for SGI InfiniteStorage.)

      SG-XPCI1FC-QL2 (single-port 2 Gb)
      SG-XPCI2FC-QF2-Z (dual-port 2 Gb)


    Note: CXFS does not automatically detect WWPNs for LSI HBAs or QLogic HBAs. See “I/O Fencing for Solaris”.


  • Any system based on UltraSPARC III, IIIi, or IV with a spare 66-MHz (or faster) PCI slot for a Fibre Channel HBA and a spare 100-Mb/s (or faster) ethernet port for the CXFS private network. CXFS supports a Solaris client only on the SPARC platform. It is not supported on other hardware platforms.

For additional latest information, see the CXFS Solaris release notes.

CXFS Commands on Solaris

The following commands are shipped as part of the CXFS Solaris package:

/usr/cxfs_cluster/bin/cxfs_client
/usr/cxfs_cluster/bin/cxfs_info
/usr/cxfs_cluster/bin/cxfsdump
/usr/sbin/grioadmin
/usr/sbin/griomon
/usr/sbin/grioqos
/usr/cxfs_cluster/bin/xvm

The cxfs_client and xvm commands are needed to include a client-only node in a CXFS cluster. The cxfs_info command reports the current status of this node in the CXFS cluster.

The pkgadd output lists all software added; see “Solaris Installation Procedure”.

For more information, see the man pages. For additional information about the GRIO commands, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and “GRIO on Solaris”.

Log Files on Solaris

The cxfs_client command creates a /var/log/cxfs_client log file. To rotate this log file, use the -z option in the /usr/cxfs_cluster/bin/cxfs_client.options file; see the cxfs_client man page for details.

For information about the log files created on server-capable administration nodes, see the CXFS Administration Guide for SGI InfiniteStorage.

CXFS Mount Scripts on Solaris

Solaris supports the CXFS mount scripts. See “CXFS Mount Scripts” in Chapter 1 and the CXFS Administration Guide for SGI InfiniteStorage.

Limitations and Considerations on Solaris

Note the following:

  • Although it is possible to mount a UFS or NFS filesystem on top of a Solaris CXFS filesystem, this is not recommended.

  • After a crash, attempts to reclaim locks and commit asynchronous writes to a CXFS filesystem from an NFS client may result in a stale file handle.

  • For optimal performance, you should set the value of the Solaris system tunable parameter maxphys in the /etc/system file. See “maxphys System Tunable for Solaris”.

  • All disk devices attached to LSI Logic HBAs must be for use only by CXFS disks; do not attach non-disk devices to any Fibre Channel HBA that is configured for CXFS use. This restriction is required because all disk devices on these HBAs (configured for CXFS) make use of the whole disk volume, which must be conveyed to Solaris via modification in the HBA driver to the value returned by the READ_CAPACITY SCSI command.

  • CXFS does not automatically detect WWPNs for LSI HBAs. See “I/O Fencing for Solaris” for instructions to set up a fencing configuration.

  • The xvm command displays duplicate entries of physvols. The number of duplicate entries correspond to the devices for each LUN.

See also Appendix B, “Filesystem and Logical Unit Specifications”.

Access Control Lists and Solaris

All CXFS files have UNIX mode bits (read, write, and execute) and optionally an access control list (ACL). For more information, see the chmod and setfacl man pages.

If you restore a CXFS file that had an ACL containing only owner-ACL entries (that is, owner/group/other/mask) from a Solaris node, upon restoration one of the following will happen:

  • When using tar(1), cpio(1), and Legato Networker: The ACL will be lost because these tools behave "intelligently" by not calling acl to set an ACL if the file has only owner/group/other/mask entries. These tools will only set the file mode. However, this does not present a change in functionality because an access permissions check on the mode and the ACL containing only owner entries will give the same result.

  • When using other backup/restore utilities: A mask will be added to the ACL if the application calls aclfor every file.

    A backup/restore utility that calls acl to set an ACL for every file will result in a file being restored with four ACL entries (that is, owner/group/other/mask), even though it may have originally had only three (that is, owner/group/other). This is due to a requirement in getfacl that it receive four ACL entries for the GETACL command to acl. (If fewer than four entries are returned, getfacl will report an error).


Note: Normally, Solaris filesystem ACLs can have up to 1024 entries for a file and a directory can have 1024 entries as well as an additional 1024 entries for the default ACL. However, CXFS filesystems on Solaris nodes in a multiOS cluster must maintain compatibility with the metadata server. The CXFS filesystems on a Solaris node are limited to a maximum of 25 ACL entries for a file and a maximum total of 50 for a directory (that is, the directory ACL plus the default ACL).

When using the ls command to display access permissions for a file with an ACL, the mode reported for a CXFS file follows Linux semantics instead of Solaris/UFS semantics.

On Solaris, a UFS file mode reports the group permission as the intersection of the GROUP and MASK entries in the ACL. If the GROUP entry is r-x and the MASK entry is rw-, the group permission will be reported as r--.

The CXFS model calls for reporting the ACL MASK for the group permission in the mode. Therefore, using the example above, the group permission will be reported as rw-. Although it appears that the group has write permission, it does not and an attempt to write to the file will be rejected. You can obtain the real (that is, effective) group permission by using the Solaris getfacl command.

maxphys System Tunable for Solaris

For optimal performance, you should set the value of the Solaris system tunable parameter maxphys in the /etc/system file. Do the following:

  1. Make a backup copy of the /etc/system file.


    Note: Exercise extreme caution in changing /etc/system and always make a backup copy.


  2. Change the value of maxphys to 0x800000 (hexadecimal) by adding the following to /etc/system :

    set maxphys=0x800000

  3. Reboot the Solaris node. This causes the change to take effect.

  4. Verify that the new value for maxphys is in effect by running the following command:

    solaris# echo "maxphys/X" | adb -k
             physmem 1f03f
             maxphys:
             maxphys:        800000

HBA Installation for Solaris

The QLogic driver is provided with Solaris 10.

The remainder of this section discusses the following:

These procedures may be performed by you or by a qualified Sun service representative. You must be logged in as root to perform the steps listed in this section.

Installing the LSI Logic HBA

To install the LSI Logic HBA, perform the following steps. Additional details are provided in the Fibre Channel to PCI-X Host Adapters User's Guide.

  1. Install the LSI Logic HBA into the Solaris system. See the chapter "Installing the Host Adapter" from the Fibre Channel to PCI-X Host Adapters User's Guide.

  2. Bring the system back up.

  3. Install the LSI Logic HBA driver software ( ITImpt, version 5.07.00 or later) according to the instructions in the driver's readme file.

    Do the following:

    1. Retrieve the driver package from the following LSI Logic website:

      http://www.lsi.com/cm/DownloadSearch.do?locale=EN

    2. Install the driver package:

      solaris# unzip itmpt-5.07.00.zip
      solaris# uncompress itmpt_install.tar.Z
      solaris# tar -xvf itmpt_install.tar
      solaris# cd install
      solaris# pkgadd -d .

    3. Install the lsi utilities package:

      solaris# uncompress lsiutils_v60.tar.Z
      solaris# tar -xvf lsiutils_v60.tar
      solaris# cd install
      solaris# pkgadd -d .

  4. For each target/LUN pair to be used by the LSI Logic HBA, use the lsiprobe utility to add entries to /kernel/drv/ssd.conf.

    For example, to add entries for targets 0 through 5 (inclusive), with each of those targets scanning LUNs 0, 2, 4, 5, and 6:

    solaris# lsiprobe -a target 0-5 lun 0,2,4-6


    Note: If you modify /kernel/drv/ssd.conf, you must reboot the system (as in step 5) in order for changes to take effect.


  5. Reboot the Solaris node:

    solaris# init 6

  6. After the system has rebooted, verify that the driver attached correctly to the HBA by following the steps “Verifying the HBA Installation”. Do not proceed until the verification succeeds.

Verifying the HBA Installation

After the system reboots, you should verify that the devices were correctly configured by running the Solaris format command. You should see a list of each device you selected.

For example:

solaris# format

Searching for disks...done

c2t200400A0B80C268Cd1: configured with capacity of 67.75GB
c2t200400A0B80C268Cd3: configured with capacity of 136.64GB

AVAILABLE DISK SELECTIONS:
       0. c0t0d0           /pci@1c,600000/scsi@2/sd@0,0
       1. c0t1d0           /pci@1c,600000/scsi@2/sd@1,0
       2. c2t200400A0B80C268Cd1           /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,1
       3. c2t200400A0B80C268Cd3           /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,3

Specify disk (enter its number):

In this example, disks 2 and 3 are being addressed by the QLogic driver, as indicated by the presence of SUNW,qlc@1 in the pathname.

You can also use the luxadm command to view the status of the HBA:

solaris# luxadm -e port
/devices/pci@1d,700000/SUNW,qlc@1/fp@0,0:devctl                    CONNECTED
/devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl                  NOT CONNECTED

solaris# luxadm probe
No Network Array enclosures found in /dev/es

Found Fibre Channel device(s):
  Node WWN:200400a0b80c268b  Device Type:Disk device
    Logical Path:/dev/rdsk/c2t200400A0B80C268Cd1s2
  Node WWN:200400a0b80c268b  Device Type:Disk device
    Logical Path:/dev/rdsk/c2t200400A0B80C268Cd3s2

The system log and console display may display warning messages similar to the following:

WARNING: /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,3 (ssd0):
        Corrupt label; wrong magic number

WARNING: /pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,1 (ssd1):
        Corrupt label; wrong magic number

For QLogic HBA, these messages means that the disk has a bad label or a DVH label, which is not supported. (QLogic HBAs support only GPT labels.)

Similar messages for an LSI Logic HBA will appear on boot for LUNs that have DVH labels. When the XVM module is loaded or started, it installs hooks into the HBA driver and automatically translates the DVH labels into SUN labels (you should not try to relabel the disks with the format command); after XVM translates the labels, you will not see these error messages.


Note: You can also use the lsiutil command to determine the number of LSI HBAs installed, the model numbers, and firmware versions.

If you are having trouble with the verification steps, see “New Storage is Not Recognized on Solaris”.

Setting Persistent Name Binding

Adding a new RAID can change the fabric name binding. To set up persistent binding, edit the /kernel/drv/itmpt.conf file and add entries of the following format, one for each target, where portWWN is the port world wide name of the RAID controller:

target-X-wwn="portWWN"

For example:

target-0-wwn="200800a0b8184c8e"
target-1-wwn="200900a0b8184c8d"
target-2-wwn="200400a0b80c268c"
target-3-wwn="200500a0b80c268c"

In this example, target-0 will be bound to the device with the port WWN 200800a0b8184c8e and the resulting devices will be:

/pci@1d,700000/IntraServer,fc@1/ssd@0,*

Preinstallation Steps for Solaris

This section provides an overview of the steps that you or a qualified Sun service representative will perform on your Solaris nodes prior to installing the CXFS software. It contains the following sections:

Adding a Private Network for Solaris Nodes

The following procedure provides an overview of the steps required to add a private network to the Solaris system. A private network is required for use with CXFS. See “Use a Private Network” in Chapter 2.

You may skip some steps, depending upon the starting conditions at your site. For details about any of these steps, see the Solaris documentation.

  1. If your system is already operational and on the network, skip to step 2.

    If your Solaris system has never been set up, bring the system to single-user mode. For example, go to the PROM prompt and boot the Solaris node into single-user mode:

    > boot -s

    As a last resort, you can reach the PROM prompt by pressing the L1-A (or Stop-A) key sequence.

  2. Edit the /etc/inet/ipnodes file so that it contains entries for each node in the cluster and its private interfaces.

    The /etc/inet/ipnodes file has the following format, where primary_hostname can be the simple hostname or the fully qualified domain name:

     IP_address    primary_hostname    aliases

    You should be consistent when using fully qualified domain names in the /etc/inet/ipnodes file. If you use fully qualified domain names on a particular node, then all of the nodes in the cluster should use the fully qualified name of that node when defining the IP/hostname information for that node in their /etc/inet/ipnodes file.

    The decision to use fully qualified domain names is usually a matter of how the clients (such as NFS) are going to resolve names for their client server programs, how their default resolution is done, and so on.

    Even if you are using the domain name service (DNS) or the network information service (NIS), you must add every IP address and hostname for the nodes to /etc/inet/ipnodes on all nodes. For example:

    190.0.2.1 server1.company.com server1
    190.0.2.3 stocks
    190.0.3.1 priv-server1
    190.0.2.2 server2.company.com server2
    190.0.2.4 bonds
    190.0.3.2 priv-server2

    You should then add all of these IP addresses to /etc/inet/ipnodes on the other nodes in the cluster.

    For more information, see the hosts, named, and nis man pages.


    Note: Exclusive use of NIS or DNS for IP address lookup for the nodes will reduce availability in situations where the NIS or DNS service becomes unreliable.

    For more information, see “Understand Hostname Resolution and Network Configuration Rules” in Chapter 2.

  3. Edit the /etc/nsswitch.conf file so that local files are accessed before either NIS or DNS. That is, the ipnodes line in /etc/nsswitch.conf must list files first.

    For example:

    ipnodes:      files nis dns 

    (The order of nis and dns is not significant to CXFS, but files must be first.)

  4. Determine the name of the private interface by using the ifconfig command as follows:

    solaris# ifconfig -a

    If the second network does not appear, it may be that a network interface card must be installed in order to provide a second network, or it may be that the network is not yet initialized.

    For example, on an Ultra Enterprise 250, the integrated Ethernet is hme0; this is the public network. The following ifconfig output shows that only the public interface exists:

    solaris# ifconfig -a
    lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000 
    hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
            inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255
            ether 8:0:20:d2:29:c5

    If the second network does not appear, do the following:

    1. If you do not have the PCI card installed, install it. Refer to your PCI documentation for instructions.

      If your card is already installed, skip to step b.

    2. Use the output from the dmesg command to determine the interface name for the private network; look for the network interface that immediately follows the public network; you may wish to search for Found. For example:

      solaris# dmesg
      
      Feb  6 09:38:36 ue250 last message repeated 42 times
      Feb  6 11:38:40 ue250 pseudo: [ID 129642 kern.info] pseudo-device: devinfo0
      Feb  6 11:38:40 ue250 genunix: [ID 936769 kern.info] devinfo0 is /pseudo/devinfo@0
      Feb  6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme0 : PCI IO 2.0 (Rev Id = c1) Found
      Feb  6 11:38:41 ue250 genunix: [ID 936769 kern.info] hme0 is /pci@1f,4000/network@1,1
      Feb  6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme1 : PCI IO 2.0 (Rev Id = c1) Found
      Feb  6 11:38:41 ue250 hme: [ID 517527 kern.info] SUNW,hme1 : Local Ethernet address = 8:0:20:cc:43:48
      Feb  6 11:38:41 ue250 pcipsy: [ID 370704 kern.info] PCI-device: SUNW,hme@1,1, hme1
      Feb  6 11:38:41 ue250 genunix: [ID 936769 kern.info] hme1 is /pci@1f,2000/SUNW,hme@1,1

      The second network is hme1; this is the private network, and is displayed after hme0 in the dmesg output. In this example, hme1 is the value needed in step c and in step 5 below.

    3. Initialize the private network's interface by using the ifconfig command as follows, where interface is the value determined in step b:

      ifconfig interface plumb

      For example:

      solaris# ifconfig hme1 plumb

      After performing the plumb, the hme1 interface will appear in the ifconfig output, although it will not contain the appropriate information (the correct information will be discovered after the system is rebooted later in step 8). For example, at this stage you would see the following:

      solaris# ifconfig -a
      lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
              inet 127.0.0.1 netmask ff000000 
      hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
              inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255
              ether 8:0:20:d2:29:c5 
      hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
              inet 0.0.0.0 netmask ff000000 broadcast 255.0.0.0
              ether 8:0:20:d2:29:c5

  5. Create a file named /etc/hostname.interface, where interface is the value determined in step 4. This file must contain the name of the private network. For example:

    solaris# cat /etc/hostname.hme1
    cxfssun3-priv


    Note: In this scenario, /etc/hostname.hme0 must contain the same value as the /etc/nodename file. For example:
    solaris# cat /etc/hostname.hme0
    cxfssun3
    solaris# cat /etc/nodename
    cxfssun3



  6. Edit the /etc/netmasks file to include the appropriate entries.

  7. (Optional) Edit the /.rhosts file if you want to use remote access or if you want to use the connectivity diagnostics provided with CXFS. Ensure that the mode of the .rhosts file is set to 600 (read and write access for the owner only).

    Make sure that the /.rhosts file on each Solaris node allows all of the nodes in the cluster to have access to each other. The connectivity tests execute a ping command from the local node to all nodes and from all nodes to the local node. To execute ping on a remote node, CXFS uses rsh as user root.

    For example, suppose you have a cluster with three nodes: admin0, solaris1, and solaris2. The /.rhosts files could be as follows (the prompt denotes the node name):

    admin0# cat /.rhosts 
    solaris1 root
    solaris1-priv root
    solaris2 root
    solaris2-priv root
     
    solaris1# cat /.rhosts
    admin0 root
    admin0-priv root
    solaris2 root
    solaris2-priv root
    
    solaris2# cat /.rhosts
    admin0 root
    admin0-priv root
    solaris1 root
    solaris1-priv root

  8. Reboot the Solaris system:

    solaris# init 6

    At this point, ifconfig will show the correct information for the private network.

    For example:

    ifconfig -a
    lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1
            inet 127.0.0.1 netmask ff000000 
    hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
            inet 128.162.2.91 netmask ffffff00 broadcast 128.162.2.255
            ether 8:0:20:d2:29:c5 
    hme1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 3
            inet 10.1.1.36 netmask ffffff00 broadcast 10.1.1.255
            ether 8:0:20:d2:29:c5

Verifying the Private and Public Networks for Solaris

For each private network on each Solaris node in the pool, verify access with the Solaris ping command. Enter the following, where nodeIPaddress is the IP address of the node:

solaris# /usr/sbin/ping -s -c 3 nodeIPaddress

For example:

solaris# /usr/sbin/ping -s -c 3 128.162.2.91
PING 128.162.2.91: 56 data bytes
64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=0. time=0. ms
64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=1. time=0. ms
64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=2. time=0. ms
64 bytes from cxfssun3.americas.sgi.com (128.162.2.91): icmp_seq=3. time=0. ms

Also execute a ping on the public networks. If ping fails, follow these steps:

  1. Verify that the network interface was configured up using ifconfig ; for example:

    solaris# /usr/sbin/ifconfig eri0
    eri0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
            inet 128.162.2.127 netmask ffffff00 broadcast 128.162.2.255
            ether 0:3:ba:d:ad:77 

    In the first output line above, UP indicates that the interface was configured up.

  2. Verify that the cables are correctly seated.

Repeat this procedure on each node.

Client Software Installation for Solaris

The CXFS software will be initially installed and configured by SGI personnel. This section provides an overview of those procedures. You can use the information in this section to verify the installation.

Solaris Installation Procedure

Installing the CXFS client software for Solaris requires approximately 20 MB of space.

To install the required software on a Solaris node, SGI personnel will do the following:

  1. Read the SGI InfiniteStorage Software Platform release notes, CXFS general release notes, and CXFS Solaris release notes in the /docs directory on the ISSP DVD and any late-breaking caveats on Supportfolio

  2. Verify that the node has been upgraded to Solaris 10 (also known as SunOS 5.10) according to the Solaris installation guide. Use the following command to display the currently installed system:

    solaris# uname -r

    This command should return a value of 5.10.

  3. Transfer the client software that was downloaded onto a server-capable administration node during its installation procedure using ftp, rcp, or scp . The location of the package on the server will be as follows:

    /usr/cluster/client-dist/CXFS_VERSION/solaris/10/noarch/SGIcxfs-sol10-vCXFS_VERSION.pkg

  4. Install the package:

    solaris# pkgadd -d SGIcxfs*.pkg

    You must select the package to be installed and then confirm that you want allow the installation of the packages with superuser permission. For example:

    solaris# pkgadd -d SGIcxfs*.pkg
    The following packages are available:
      1  SGIcxfs     SGI CXFS client software
                     (sparc) 5.0.0.3
    
    Select package(s) you wish to process (or 'all' to process
    all packages). (default: all) [?,??,q]: 1
    
    Processing package instance <SGIcxfs> from
    </tmp/SGIcxfs-sol10-v5.0.0.3.pkg>
    
    SGI CXFS client software(sparc) 5.0.0.3
    /*
     *  Copyright (c) 2008 Silicon Graphics, Inc.  ALL RIGHTS RESERVED
    
    ...
     */
    Using </> as the package base directory.
    ## Processing package information.
    ## Processing system information.
       3 package pathnames are already properly installed.
    ## Verifying disk space requirements.
    ## Checking for conflicts with packages already installed.
    ## Checking for setuid/setgid programs.
    
    This package contains scripts which will be executed with super-user
    permission during the process of installing this package.
    
    Do you want to continue with the installation of <SGIcxfs> [y,n,?] y
    
    Installing SGI CXFS client software as <SGIcxfs>
    
    ## Installing part 1 of 1.
    /etc/init.d/cxfs_cluster
    /etc/rc0.d/K28cxfs_cluster <symbolic link>
    /etc/rc1.d/K28cxfs_cluster <symbolic link>
    /etc/rc2.d/S77cxfs_cluster <symbolic link>
    /etc/rc3.d/S77cxfs_cluster <symbolic link>
    /etc/rcS.d/K28cxfs_cluster <symbolic link>
    /usr/cxfs_cluster/bin/cxfs_client
    /usr/cxfs_cluster/bin/cxfs_info
    /usr/cxfs_cluster/bin/cxfsdump
    /usr/cxfs_cluster/bin/cxfslicense
    /usr/cxfs_cluster/bin/frametest
    /usr/cxfs_cluster/bin/xvm
    /usr/kernel/drv/cell.conf
    /usr/kernel/drv/sparcv9/cell
    /usr/kernel/drv/sparcv9/xvm
    /usr/kernel/drv/xvm.conf
    /usr/kernel/fs/sparcv9/cxfs
    /usr/kernel/misc/sparcv9/hba_hook
    /usr/lib/sparcv9/libgrio.so
    /usr/sbin/clmount
    /usr/sbin/grioadmin
    /usr/sbin/griomon
    /usr/sbin/grioqos
    /usr/sbin/idbg
    /usr/share/man/man1/xvm.1
    /usr/share/man/man1m/cxfs_client.1m
    /usr/share/man/man1m/cxfs_info.1m
    /usr/share/man/man1m/frametest.1m
    /var/cluster/cxfs_client-scripts/cxfs-post-mount
    /var/cluster/cxfs_client-scripts/cxfs-post-umount
    /var/cluster/cxfs_client-scripts/cxfs-pre-mount
    /var/cluster/cxfs_client-scripts/cxfs-pre-umount
    /var/cluster/cxfs_client-scripts/cxfs-reprobe
    [ verifying class <none> ]
    ## Executing postinstall script.
    Starting CXFS services...
    cxfs_client daemon started

Verifying the Solaris Installation

To verify that the CXFS software has been installed properly, use the pkginfo command as follows:

pkginfo -l SGIcxfs

For example, the following output indicates that the CXFS package installed properly:

solaris# pkginfo -l SGIcxfs
   PKGINST:  SGIcxfs
      NAME:  SGI CXFS client software
  CATEGORY:  system
      ARCH:  sparc
   VERSION:  5.0.0.3
   BASEDIR:  /
    VENDOR:  Silicon Graphics Inc.
    PSTAMP:  cxfssun120080213222059
  INSTDATE:  Feb 14 2008 07:15
    STATUS:  completely installed
     FILES:       38 installed pathnames
                   5 directories
                  22 executables
               20179 blocks used (approx)

I/O Fencing for Solaris

I/O fencing is required on Solaris nodes in order to protect data integrity of the filesystems in the cluster.

The /etc/fencing.conf file enumerates the WWPN for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for the HBA WWPN as a 64-bit hexadecimal number.


Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric.

If used, /etc/fencing.conf must contain a simple list of WWPNs, one per line. You must update it whenever the HBA configuration changes, including the replacement of an HBA.

Do the following:

  1. Set up the switch and HBA. See the release notes for supported hardware.

  2. Determine the HBA WWNs:

    • If the HBA is a Sun QLogic card, use the /usr/sbin/fcinfo command. For example:

      # /usr/sbin/fcinfo hba-port | grep 'HBA Port WWN:' | cut -d':' -f2
       210000e08b86d53c
       210100e08ba6d53c

      For more information, see the QLogic documentation.

    • If the HBA is an LSI card, you can use the lsiutil command to scan for devices. For example for ports 1 and 2:

      # lsiutil -p 1 8 | grep 'FC949X Port' | awk '{print $3}'
      100000062b114f50
      # lsiutil -p 2 8 | grep 'FC949X Port' | awk '{print $3}'
      100000062b114f51

      For more information, see the LSI documentation.

    • If either of the above do not work, do the following:

      1. Follow the Fibre Channel cable on the back of the node to determine the port to which it is connected in the switch. Ports are numbered beginning with 0. (For example, if there are 8 ports, they will be numbered 0 through 7.)

      2. Connect to the switch via telnet and log in as user admin. (On Brocade switches, the password is password by default).

      3. Use switchshow command. For example:

        brocade04:admin> switchshow
        switchName:     brocade04
        switchType:     2.4
        switchState:    Online
        switchRole:     Principal
        switchDomain:   6
        switchId:       fffc06
        switchWwn:      10:00:00:60:69:12:11:9e
        switchBeacon:   OFF
        port  0: sw  Online        F-Port  20:00:00:01:73:00:2c:0b
        port  1: cu  Online        F-Port  21:00:00:e0:8b:02:36:49
        port  2: cu  Online        F-Port  21:00:00:e0:8b:02:12:49
        port  3: sw  Online        F-Port  20:00:00:01:73:00:2d:3e
        port  4: cu  Online        F-Port  21:00:00:e0:8b:02:18:96
        port  5: cu  Online        F-Port  21:00:00:e0:8b:00:90:8e
        port  6: sw  Online        F-Port  20:00:00:01:73:00:3b:5f
        port  7: sw  Online        F-Port  20:00:00:01:73:00:33:76
        port  8: sw  Online        F-Port  21:00:00:e0:8b:01:d2:57
        port  9: sw  Online        F-Port  21:00:00:e0:8b:01:0c:57
        port 10: sw  Online        F-Port  20:08:00:a0:b8:0c:13:c9
        port 11: sw  Online        F-Port  20:0a:00:a0:b8:0c:04:5a
        port 12: sw  Online        F-Port  20:0c:00:a0:b8:0c:24:76
        port 13: sw  Online        L-Port  1 public
        port 14: sw  No_Light
        port 15: cu  Online        F-Port  21:00:00:e0:8b:00:42:d8

        The WWPN is the hexadecimal string to the right of the port number. For example, the WWPN for port 0 is 2000000173002c0b (you must remove the colons from the WWPN reported in the switchshow output to produce the string to be used in the fencing file).

  3. Edit or create /etc/fencing.conf and add the WWPN for the port determined in step 2. (Comment lines begin with #.)

    For dual-ported HBAs, you must include the WWPNs of any ports that are used to access cluster disks. This may result in multiple WWPNs per HBA in the file; the numbers will probably differ by a single digit.

    For example, if you determined that port 0 is the port connected to the switch, your fencing file should contain the following:

    # WWPN of the HBA installed on this system
    #
    2000000173002c0b

  4. To configure fencing, see the CXFS Administration Guide for SGI InfiniteStorage.

Start/Stop cxfs_client for Solaris

The /etc/init.d/cxfs_cluster script will be invoked automatically during normal system startup and shutdown procedures. This script starts and stops the cxfs_client daemon.

To start cxfs_client manually, enter the following:

solaris# /etc/init.d/cxfs_cluster start

To stop cxfs_client manually, enter the following:

solaris# /etc/init.d/cxfs_cluster stop

To stop and then start cxfs_client manually, enter the following:

solaris# /etc/init.d/cxfs_cluster restart

Maintenance for Solaris

This section contains the following:

Updating the CXFS Software for Solaris


Note: Before upgrading CXFS software, ensure that no applications on the node are accessing files on a CXFS filesystem.

To upgrade CXFS on a Solaris system, do the following:

  1. Remove the current package:

    solaris# pkgrm SGIcxfs
    
    The following package is currently installed:
       SGIcxfs         SGI CXFS client software
                       (sparc) releaselevel
    
    Do you want to remove this package? [y,n,?,q] y
    
    # Removing installed package instance <SGIcxfs>
    
    This package contains scripts which will be executed with super-user
    permission during the process of removing this package.
    
    Do you want to continue with the removal of this package [y,n,?,q] y
    # Verifying package dependencies
    ...

  2. Reboot the Solaris system:

    solaris# reboot

  3. Obtain the CXFS update software from Supportfolio according to the directions in CXFS Administration Guide for SGI InfiniteStorage

  4. Follow the installation instructions to install the new package. See “Client Software Installation for Solaris”.


Modifying the CXFS Software for Solaris

You can modify the behavior of the CXFS client daemon ( cxfs_client) by placing options in the /usr/cxfs_cluster/bin/cxfs_client.options file. The available options are documented in the cxfs_client man page.


Caution: Some of the options are intended to be used internally by SGI only for testing purposes and do not represent supported configurations. Consult your SGI service representative before making any changes.

To see if cxfs_client is using the options in cxfs_client.options, enter the following:

solaris# ps -ef | grep cxfs

Recognizing Storage Changes for Solaris

If you make changes to your storage configuration, you must rerun the HBA utilities to reprobe the storage. See “HBA Installation for Solaris”.

GRIO on Solaris

CXFS supports guaranteed-rate I/O (GRIO) version 2 on the Solaris platform. Application bandwidth reservations must be explicitly released by the application before exit. If the application terminates unexpectedly or is killed, its bandwidth reservations are not automatically released and will cause a bandwidth leak. If this happens, the lost bandwidth could be recovered by rebooting the client node.

For more information, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and the Guaranteed-Rate I/O Version 2 Guide.

XVM Failover V2 on Solaris

Following is an example of the /etc/failover2.conf file on Solaris using a QLogic HBA:

/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200400a0b80c268c,1 affinity=1 preferred
/pci@1d,700000/SUNW,qlc@1,1/fp@0,0/ssd@w200400a0b80c268c,1 affinity=1
/pci@1d,700000/SUNW,qlc@1/fp@0,0/ssd@w200500a0b80c268c,1 affinity=2
/pci@1d,700000/SUNW,qlc@1,1/fp@0,0/ssd@w200500a0b80c268c,1 affinity=2

In this case:

  • SUNW,qlc@1 is the first port on the PCI card

  • SUNW,qlc@1,1 is the second port on the PCI card

  • 200400a0b80c268c is controller A on the TP9XXX

  • 200500a0b80c268c is controller B on the TP9XXX

Following is an example using an LSI HBA:

<XVM physvol phys/cc_is4000-lun0>
     pci@1f,2000/IntraServer,fc@1,1/ssd@0,0 <dev 130> affinity=1
     pci@1f,2000/IntraServer,fc@1,1/ssd@2,0 <dev 146> affinity=0 preferred

<XVM physvol phys/cc_is4000-lun1>
     pci@1f,2000/IntraServer,fc@1,1/ssd@0,1 <dev 738> affinity=1 preferred
     pci@1f,2000/IntraServer,fc@1,1/ssd@2,1 <dev 930> affinity=0

<XVM physvol phys/cc_is4000-lun2>
     pci@1f,2000/IntraServer,fc@1,1/ssd@0,2 <dev 746> affinity=1
     pci@1f,2000/IntraServer,fc@1,1/ssd@2,2 <dev 938> affinity=0 preferred

<XVM physvol phys/cc_is4000-lun3>
     pci@1f,2000/IntraServer,fc@1,1/ssd@0,3 <dev 754> affinity=1 preferred
     pci@1f,2000/IntraServer,fc@1,1/ssd@2,3 <dev 946> affinity=0

For more information, see “XVM Failover and CXFS” in Chapter 1, the comments in the /etc/failover2.conf file, CXFS Administration Guide for SGI InfiniteStorage, and the XVM Volume Manager Administrator's Guide.

Troubleshooting for Solaris

This section contains the following:

For general troubleshooting information, see Chapter 11, “General Troubleshooting” and Appendix D, “Error Messages”.

The cxfs_client Daemon is Not Started on Solaris

Confirm that the cxfs_client is not running. The following command would list the cxfs_client process if it were running:

solaris# ps -ef | grep cxfs_client

Check the cxfs_client log file for errors.

Restart cxfs_client as described in “Start/Stop cxfs_client for Linux” in Chapter 5 and watch the cxfs_client log file for errors.

Filesystems Do Not Mount on Solaris

If cxfs_info reports that cms is up but XVM or the filesystem is in another state, then one or more mounts is still in the process of mounting or has failed to mount.

The CXFS node might not mount filesystems for the following reasons:

  • The client may not be able to see all the LUNs. This is usually caused by misconfiguration of the HBA or the SAN fabric:

    • Can the HBA see all of the LUNs for the filesystems it is mounting?

    • Can the operating system kernel see all of the LUN devices?

    See “New Storage is Not Recognized on Solaris”.

  • The cxfs_client daemon may not be running. See “The cxfs_client Daemon is Not Started on Solaris ”.

  • The filesystem may have an unsupported mount option. Check the cxfs_client.log for mount option errors or any other errors that are reported when attempting to mount the filesystem.

  • The cluster membership (cms), XVM, or the filesystems may not be up on the node. Execute the /usr/cxfs_cluster/bin/cxfs_info command to determine the current state of cms, XVM, and the filesystems. If the node is not up for each of these, then check the /var/log/cxfs_client log to see what actions have failed.

    Do the following:

    • If cms is not up, check the following:

    • If XVM is not up, check that the HBA is active and can see the LUNs.

    • If the filesystem is not up, check that one or more filesystems are configured to be mounted on this node and check the /var/log/cxfs_client file for mount errors.

New Storage is Not Recognized on Solaris

If you have a problem with an HBA, verify that you enabled fabric mode. See “Recognizing Storage Changes for Solaris”.

Large Log Files on Solaris

The /var/log/cxfs_client log file may become quite large over a period of time if the verbosity level is increased. To manually rotate this log file, use the -z option in the /usr/cxfs_cluster/bin/cxfs_client.options file.

For information about the log files created on server-capable administration nodes, see the CXFS Administration Guide for SGI InfiniteStorage.

Changing the CXFS Heartbeat Value on Solaris

To view the CXFS heartbeat value on Solaris, use the following:

# echo mtcp_hb_period/D | adb -k
physmem 3df86
mtcp_hb_period:
mtcp_hb_period: 600

Using the -k option to the adb(1) debugger causes it to attach to a live kernel. Echoing the command allows you to put it on a single line.

For example, to reset the value to 15 seconds, enter the following (the value is in Hz):

# echo mtcp_hb_period/W0t1500 | adb -kw
physmem 3df86
mtcp_hb_period: 0x258           =       0x5dc

Reporting Solaris Problems

When reporting a problem about a CXFS Solaris node to SGI, you should retain the following information:

  • If there is a system panic, retain the system core file in /var/crash/hostname on a Solaris node.

  • Output from the crash utility.

  • mdb(1M) modular debugger output:

    • For panics or generated dumps, use the following commands and save the output:

      $c (or $C)
      $r
      $<msgbuf

    • For dumps from hangs:

      $<threadlist
      $c (or $C)
      $r
      $<msgbuf

  • A list of the installed CXFS packages. Use the pkginfo command as follows:

    # pkginfo -l SGIcxfs

  • A list of the Solaris patches that have been installed. Use the showrev command. The showrev command without options prints a summary and the -p option lists the revision information about patches.

  • A list of the loaded Solaris kernel modules and versions. Use the modinfo command.

  • Output about the cluster obtained from the cxfsdump utility run on a server-capable administration node. When run in local mode on a Solaris node, it stores information in /var/cluster/cxfsdump-data/ nodename.tar.gz.

  • Output from the LSI /usr/sbin/lsiutil command, which displays the number of LSI HBAs installed, the model numbers, and firmware versions.

If any of the above Solaris tools are not currently installed on your Solaris system, you should install them.