Chapter 7. SGI ProPack Client-Only Platform

SGI ProPack for Linux is an overlay product that adds or enhances features in the supported Linux base distributions. This chapter discusses the following:

For information about system tunable parameters, see CXFS Administration Guide for SGI InfiniteStorage.

For information about SGI ProPack server-capable administration nodes, see CXFS Administration Guide for SGI InfiniteStorage.

CXFS on SGI ProPack Client-Only Nodes

This section discusses the following:

Requirements for SGI ProPack Client-Only Nodes

In addition to the items listed in “Requirements” in Chapter 1, using an SGI ProPack client-only node to support CXFS requires the following hardware and software specified in the release notes:

  • SGI ProPack software (see the release notes for specific kernel information) for SGI Altix and SGI Altix XE:

    • SGI ProPack 5 SP2 (SLES)

    • SGI ProPack 5 SP3 (SLES)

    • SGI ProPack 5 SP4 (SLES)

    • SGI ProPack 5 SP5 (SLES)

    For more information about using SGI ProPack 5 SP 2 as a server-capable administration node, see the CXFS Administration Guide for SGI InfiniteStorage. Nodes that you intend to run as metadata servers must be installed as server-capable administration nodes; all other nodes should be client-only nodes.

  • A supported SAN hardware configuration.


    Note: For details about supported hardware, see the Entitlement Sheet that accompanies the release materials. Using unsupported hardware constitutes a breach of the CXFS license.


  • Use a network switch. (A network hub is not supported.) The switch should be at least 100baseT.

  • A private 100baseT or Gigabit Ethernet TCP/IP network connected to each node. When using Gigabit Ethernet, do not use jumbo frames. For more information, see the tgconfig man page.

  • Serial lines and/or supported Fibre Channel switches. For supported switches, see the release notes. Either system reset or I/O fencing is required for all nodes.

  • At least one host bus adapter:

    • QLogic QLA2310, QLA2342, or QLA2344

    • LSI Logic LSI7104XP-LC, LSI7204XP-LC, or LSI7204EP-LC


      Note: The LSI HBA requires the 01030600 firmware.


  • RAID hardware as specified in the release notes.

  • The XVM volume manager, which is provided with the CXFS release.

  • If you use I/O fencing and ipfilterd on a node, the ipfilterd configuration must allow communication between the node and the telnet port on the switch.

CXFS Commands on SGI ProPack Client-Only Nodes

The following commands are shipped as part of the CXFS SGI ProPack package:

/usr/cluster/bin/cxfs-config
/usr/cluster/bin/cxfs_client
/usr/cluster/bin/cxfs_info
/usr/cluster/bin/cxfscp
/usr/cluster/bin/cxfsdump
/usr/sbin/grioadmin
/usr/sbin/grioqos
/sbin/xvm

The cxfs_client and xvm commands are needed to include a client-only node in a CXFS cluster. The cxfs_info command reports the current status of this node in the CXFS cluster. For more information, see the man pages.

Log Files on SGI ProPack Client-Only Nodes

You should monitor the /var/log/cxfs_client and /var/log/messages log files for problems.

Look for a Membership delivered message to indicate that a cluster was formed.

The Linux platform uses the logrotate system utility to rotate the cxfs_client logs:

  • The /etc/logrotate.conf file specifies how often system logs are rotated

  • The /etc/logrotate.d/cxfs_client file specifies the manner in which cxfs_client logs are rotated

CXFS Mount Scripts on SGI ProPack Client-Only Nodes

SGI ProPack supports the CXFS mount scripts. See “CXFS Mount Scripts” in Chapter 1 and the CXFS Administration Guide for SGI InfiniteStorage.

Limitations and Considerations for SGI ProPack Client-Only Nodes

The following sections highlight limitations and considerations for client-only SGI ProPack nodes.


Note: For limitations and considerations for any SGI ProPack node, see the information in CXFS Administration Guide for SGI InfiniteStorage.


  • On systems running SUSE Linux Enterprise Server 10 (SLES 10) that are greater than 64 CPUs, there are issues with using the md driver and CXFS. The md driver holds the BKL (Big Kernel Lock), which is a single, system-wide spin lock. Attempting to acquire this lock can add substantial latency to a driver's operation, which in turn holds off other processes such as CXFS. The delay causes CXFS to lose membership. This problem has been observed specifically when an md pair RAID split is done, such as the following:

    raidsetfaulty /dev/md1 /dev/path/to/partition

Client Software Installation for SGI ProPack Client-Only Nodes

The CXFS client-only software will be initially installed and configured by SGI personnel. This section provides an overview of those procedures. You can use the information in this section to verify the installation.


Note: Package version numbers shown here are examples; your installed system may differ.

This section covers the following:

SGI ProPack Client-Only Installation Procedure

Installing the CXFS client software for SGI ProPack requires approximately 50-200 MB of space, depending upon the packages installed at your site.

To install the required software on an SGI ProPack client-only node, SGI personnel will do the following:

  1. Read the SGI InfiniteStorage Software Platform release notes, CXFS general release notes, and CXFS SGI ProPack release notes in the /docs directory on the ISSP DVD and any late-breaking caveats on Supportfolio to learn about any changes in the installation procedure.

  2. Confirm that you have installed SLES 10 SP1. (See the /etc/SuSE-release file.) Ensure that you have the appropriate kernel as listed in the SGI InfiniteStorage Software Platform release notes.

  3. Confirm that you have installed SGI ProPack 5 SP 5. (See the /etc/sgi-release file.) For more information, see the SGI ProPack 5 for Linux Service Pack 5 Start Here.

  4. Install any required patches. See the SGI ProPack releasenotes/README file for more information.

  5. If you had to install software in one of the above steps, reboot the system:

    propack# /sbin/reboot

  6. Transfer the client software that was downloaded onto a server-capable administration node during its installation procedure using ftp, rcp, or scp . The location of the tarball on the server will be as follows:

    /usr/cluster/client-dist/CXFS_VERSION/linux/sles10sp1/ARCHITECTURE/cxfs-client.tar.gz

    For example, for an Altix ia64 server:

    /usr/cluster/client-dist/5.0.0.3/linux/sles10sp1/ia64/cxfs-client.tar.gz

  7. Disassemble the downloaded tarball:

    propack# tar -zxvf tarball

  8. Install the CXFS software:

    • Including GRIOv2:

      propack# rpm -Uvh *.rpm

    • Without GRIOv2:

      propack# rpm -Uvh cxfs*rpm sgi*rpm

  9. Edit the /etc/cluster/config/cxfs_client.options file as necessary. See the cxfs_client (1M) man page.

  10. Reboot the system with the newly installed kernel:

    propack# reboot

Installing the Performance Co-Pilot Agent

The cxfs_utils package includes a Performance Co-Pilot (PCP) agent for monitoring CXFS heartbeat, CMS status and other statistics. If you want to use this feature, you must also install the following PCP packages:

pcp-open
pcp-sgi

These packages and are included on the first and second SGI ProPack CDs (respectively). You can obtain the open source PCP package from ftp://oss.sgi.com/projects/pcp/download

Verifying the SGI ProPack Client-Only Installation

To verify that the CXFS software has been installed properly, use the rpm -q command to query the packages.

To verify the SGI ProPack release, display the /etc/sgi-release file.

I/O Fencing for SGI ProPack Client-Only Nodes

On the SGI ProPack platform, the cxfs_client software automatically detects the world wide port names (WWPNs) of any supported host bus adapters (HBAs) in the system that are connected to a switch that is configured in the cluster database. These HBAs will then be available for fencing.

However, if no WWPNs are detected, there will be messages logged to the /var/log/cxfs_client file.

If no WWPNs are detected, you can manually specify the WWPNs in the /etc/fencing.conf fencing file.


Note: This method does not work if the WWPNs are partially discovered.

The /etc/fencing.conf file enumerates the WWPN for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for the HBA WWPN as a 64-bit hexadecimal number.


Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric.

If used, /etc/fencing.conf must contain a simple list of WWPNs, one per line. You must update it whenever the HBA configuration changes, including the replacement of an HBA.

Do the following:

  1. Set up the switch and HBA. See the release notes for supported hardware.

  2. Determine the HBA WWPN: Follow the Fibre Channel cable on the back of the node to determine the port to which it is connected in the switch. Ports are numbered beginning with 0. (For example, if there are 8 ports, they will be numbered 0 through 7.)

  3. Use the telnet command to connect to the switch and log in as user admin. (On Brocade switches, the password is password by default).

  4. Execute the switchshow command to display the switches and their WWPN numbers.

    For example:

    brocade04:admin> switchshow
    switchName:     brocade04
    switchType:     2.4
    switchState:    Online   
    switchRole:     Principal
    switchDomain:   6
    switchId:       fffc06
    switchWwn:      10:00:00:60:69:12:11:9e
    switchBeacon:   OFF
    port  0: sw  Online        F-Port  20:00:00:01:73:00:2c:0b
    port  1: cu  Online        F-Port  21:00:00:e0:8b:02:36:49
    port  2: cu  Online        F-Port  21:00:00:e0:8b:02:12:49
    port  3: sw  Online        F-Port  20:00:00:01:73:00:2d:3e
    port  4: cu  Online        F-Port  21:00:00:e0:8b:02:18:96
    port  5: cu  Online        F-Port  21:00:00:e0:8b:00:90:8e
    port  6: sw  Online        F-Port  20:00:00:01:73:00:3b:5f
    port  7: sw  Online        F-Port  20:00:00:01:73:00:33:76
    port  8: sw  Online        F-Port  21:00:00:e0:8b:01:d2:57
    port  9: sw  Online        F-Port  21:00:00:e0:8b:01:0c:57
    port 10: sw  Online        F-Port  20:08:00:a0:b8:0c:13:c9
    port 11: sw  Online        F-Port  20:0a:00:a0:b8:0c:04:5a
    port 12: sw  Online        F-Port  20:0c:00:a0:b8:0c:24:76
    port 13: sw  Online        L-Port  1 public
    port 14: sw  No_Light      
    port 15: cu  Online        F-Port  21:00:00:e0:8b:00:42:d8

    The WWPN is the hexadecimal string to the right of the port number. For example, the WWPN for port 0 is 2000000173002c0b (you must remove the colons from the WWPN reported in the switchshow output to produce the string to be used in the fencing file).

  5. Edit or create /etc/fencing.conf and add the WWPN for the port determined in step 2. (Comment lines begin with #.)

    For dual-ported HBAs, you must include the WWPNs of any ports that are used to access cluster disks. This may result in multiple WWPNs per HBA in the file; the numbers will probably differ by a single digit.

    For example, if you determined that port 0 is the port connected to the switch, your fencing file should contain the following:

    # WWPN of the HBA installed on this system
    #
    2000000173002c0b

  6. To configure fencing, see the CXFS Administration Guide for SGI InfiniteStorage.

Start/Stop cxfs_client for SGI ProPack Client-Only Nodes

The cxfs_cluster service will be invoked automatically during normal system startup and shutdown procedures. This script starts and stops the cxfs_client daemon.

  • To start cxfs_client manually, enter the following:

    propack# service cxfs_client start
    Loading cxfs modules:                                      [  OK  ]
    Mounting devfs filesystems:                                [  OK  ]
    Starting cxfs client:                                      [  OK  ]

  • To stop CXFS processes manually, enter the following command:

    propack# service cxfs_client stop
    Stopping cxfs client:                                      [  OK  ]

Maintenance for SGI ProPack Client-Only Nodes

This section discusses the following:

Modifying the CXFS Software for SGI ProPack

You can modify the CXFS client daemon (/usr/cluster/bin/cxfs_client ) by placing options in the cxfs_client.options file:

/etc/cluster/config/cxfs_client.options

The available options are documented in the cxfs_client man page.


Caution: Some of the options are intended to be used internally by SGI only for testing purposes and do not represent supported configurations. Consult your SGI service representative before making any changes.

For example, to see if cxfs_client is using the options in cxfs_client.options, enter the following:

propack# ps -ax | grep cxfs_client
 3612 ?        S      0:00 /usr/cluster/bin/cxfs_client -i cxfs3-5
 3841 pts/0    S      0:00 grep cxfs_client

Recognizing Storage Changes for SGI ProPack

When cxfs_client needs to rescan disk buses, it executes the /var/cluster/cxfs_client-scripts/cxfs-reprobe script. This requires the use of parameters in SGI ProPack due to limitations in the Linux SCSI layer. You can export these parameters from the /etc/cluster/config/cxfs_client.options file.

The script detects the presence of the SCSI and/or XSCSI layers on the system and defaults to probing whichever layers are detected. You can override this decision by setting CXFS_PROBE_SCSI and/or CXFS_PROBE_XSCSI to either 0 (to disable the probe) or 1 (to force the probe) on the appropriate bus.

When an XSCSI scan is performed, all buses are scanned by default. You can override this decision by specifying a space-separated list of buses in CXFS_PROBE_XSCSI_BUSES. (If you include space, you must enclose the list within single quotation marks.) For example, for SGI ProPack 4:

export CXFS_PROBE_XSCSI_BUSES='/dev/xscsi/pci01.03.0-1/bus /dev/xscsi/pci02.01.0-2/bus'

For SGI ProPack 5:

export CXFS_PROBE_XSCSI_BUSES='/dev/xscsi/pci0001:00:03.0-1/bus /dev/xscsi/pci0002:00:01.0-2/bus'

When a SCSI scan is performed, a fixed range of buses/channels/IDs and LUNs are scanned; these ranges may need to be changed to ensure that all devices are found. The ranges can also be reduced to increase scanning speed if a smaller space is sufficient.

The following summarizes the environment variables (separate multiple values by white space and enclose withing single quotation marks):

CXFS_PROBE_SCSI=0|1
 

Stops (0) or forces (1) a SCSI probe. Default: 1 if SCSI

CXFS_PROBE_SCSI_BUSES=BusList
 

Scans the buses listed. Default: 0 1 2

CXFS_PROBE_SCSI_CHANNELS=ChannelList
 

Scans the channels listed. Default: 0

CXFS_PROBE_SCSI_IDS=IDList
 

Scans the IDS listed. Default: 0 1 2 3

CXFS_PROBE_SCSI_LUNS=LunList
 

Scans the LUNs listed. Default: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

CXFS_PROBE_XSCSI=0|1
 

Stops (1) or forces (1) an XSCSI probe. Default: 1 if XSCSI

CXFS_PROBE_XSCSI_BUSES=BusList
 

Scans the buses listed. Default: all XSCSI buses

For example, the following would only scan the first two SCSI buses:

export CXFS_PROBE_SCSI_BUSES='0 1'

The following would scan 16 LUNs on each bus, channel, and ID combination (all on one line):

export CXFS_PROBE_SCSI_LUNS='0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15'

Other options within the /etc/cluster/config/cxfs_client.options file begin with a - character. Following is an example cxfs_client.options file:

propack# Example cxfs_client.options file
#
-Dnormal -serror
export CXFS_PROBE_SCSI_BUSSES=1
export CXFS_PROBE_SCSI_LUNS='0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20'


Note: The - character or the term export must start in the first position of each line in the cxfs_client.options file; otherwise, they are ignored by the cxfs_client service.


GRIO on SGI ProPack Client-Only Nodes

CXFS supports guaranteed-rate I/O (GRIO) version 2 on the SGI ProPack client-only platform.

GRIO is disabled by default on SGI ProPack. To enable GRIO, change the following line in /etc/cluster/config/cxfs_client.options from:

export GRIO2=off

to:

export GRIO2=on

Application bandwidth reservations must be explicitly released by the application before exit. If the application terminates unexpectedly or is killed, its bandwidth reservations are not automatically released and will cause a bandwidth leak. If this happens, the lost bandwidth could be recovered by rebooting the client node.

An SGI ProPack client-only node can mount a GRIO-managed filesystem and supports application- and node-level reservations. An SGI ProPack client-only node client will interoperate with the dynamic bandwidth allocator for all I/O outside of any reservation.

For more information, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and the Guaranteed-Rate I/O Version 2 Guide.

XVM Failover V2 on SGI ProPack Client-Only Nodes

Following is an example of the /etc/failover2.conf file on SGI ProPack:

/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port1/lun4/disc, affinity=1
/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port2/lun4/disc, affinity=2 
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port1/lun4/disc, affinity=1
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port2/lun4/disc, affinity=2
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port1/lun4/disc, affinity=4
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port2/lun4/disc, affinity=3 preferred
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port1/lun4/disc, affinity=4
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port2/lun4/disc, affinity=3

For more information, see “XVM Failover and CXFS” in Chapter 1, the comments in the /etc/failover2.conf.example file, CXFS Administration Guide for SGI InfiniteStorage, and the XVM Volume Manager Administrator's Guide.

Reporting SGI ProPack Client-Only Nodes Problems

Retain the following information for SGI ProPack nodes:

  • The kernel you are running:

    propack# uname -a

  • The CXFS packages you are running:

    propack# rpm -q cxfs_client sgi-cxfs-kmp cxfs_utils cxfs-xvm-cmds

  • The number and types of processors in your machine:

    propack# cat /proc/cpuinfo

  • The hardware installed on your machine:

    propack# /sbin/lspci

  • Modules that are loaded on your machine:

    propack# /sbin/lsmod

  • The /var/log/cxfs_client log file

  • Any messages that appeared in the system logs immediately before the system exhibited the problem.

  • Output about the cluster obtained from the cxfsdump utility run on a server-capable administration node.

  • After a system kernel panic, the debugger information from the KDB built-in kernel debugger.

  • Fibre Channel HBA World Wide name mapping:

    cat /sys/class/fc_transport/bus_ID/node_name

    For example:

    cat /sys/class/fc_transport/11:0:0:0/node_name

    The bus_ID value is the output of hwinfo --disk in the SysFS BusID field.

  • Output from the following commands:

    • Information from the following files:

      /var/log/messages
      /var/log/cxfs_client     (for client-only nodes)
      /etc/failover.conf       (for XVM failover version 1)
      /etc/failover2.conf      (for XVM failover version 2)
      /etc/hosts
      /proc/discontig 

    • Output from the following commands:

      /usr/cluster/bin/cdbutil gettree '#'
      /usr/bin/hinv
      /usr/bin/topology
      /sbin/xvm show -v phys
      /sbin/xvm show -top -v vol
      /bin/netstat -ia