SGI ProPack for Linux is an overlay product that adds or enhances features in the supported Linux base distributions. This chapter discusses the following:
For information about system tunable parameters, see CXFS Administration Guide for SGI InfiniteStorage.
For information about SGI ProPack server-capable administration nodes, see CXFS Administration Guide for SGI InfiniteStorage.
This section discusses the following:
In addition to the items listed in “Requirements” in Chapter 1, using an SGI ProPack client-only node to support CXFS requires the following hardware and software specified in the release notes:
SGI ProPack software (see the release notes for specific kernel information) for SGI Altix and SGI Altix XE:
SGI ProPack 5 SP2 (SLES)
SGI ProPack 5 SP3 (SLES)
SGI ProPack 5 SP4 (SLES)
SGI ProPack 5 SP5 (SLES)
For more information about using SGI ProPack 5 SP 2 as a server-capable administration node, see the CXFS Administration Guide for SGI InfiniteStorage. Nodes that you intend to run as metadata servers must be installed as server-capable administration nodes; all other nodes should be client-only nodes.
A supported SAN hardware configuration.
![]() | Note: For details about supported hardware, see the Entitlement Sheet that accompanies the release materials. Using unsupported hardware constitutes a breach of the CXFS license. |
Use a network switch. (A network hub is not supported.) The switch should be at least 100baseT.
A private 100baseT or Gigabit Ethernet TCP/IP network connected to each node. When using Gigabit Ethernet, do not use jumbo frames. For more information, see the tgconfig man page.
Serial lines and/or supported Fibre Channel switches. For supported switches, see the release notes. Either system reset or I/O fencing is required for all nodes.
At least one host bus adapter:
QLogic QLA2310, QLA2342, or QLA2344
LSI Logic LSI7104XP-LC, LSI7204XP-LC, or LSI7204EP-LC
![]() | Note: The LSI HBA requires the 01030600 firmware. |
RAID hardware as specified in the release notes.
The XVM volume manager, which is provided with the CXFS release.
If you use I/O fencing and ipfilterd on a node, the ipfilterd configuration must allow communication between the node and the telnet port on the switch.
The following commands are shipped as part of the CXFS SGI ProPack package:
/usr/cluster/bin/cxfs-config |
/usr/cluster/bin/cxfs_client |
/usr/cluster/bin/cxfs_info |
/usr/cluster/bin/cxfscp |
/usr/cluster/bin/cxfsdump |
/usr/sbin/grioadmin |
/usr/sbin/grioqos |
/sbin/xvm |
The cxfs_client and xvm commands are needed to include a client-only node in a CXFS cluster. The cxfs_info command reports the current status of this node in the CXFS cluster. For more information, see the man pages.
You should monitor the /var/log/cxfs_client and /var/log/messages log files for problems.
Look for a Membership delivered message to indicate that a cluster was formed.
The Linux platform uses the logrotate system utility to rotate the cxfs_client logs:
The /etc/logrotate.conf file specifies how often system logs are rotated
The /etc/logrotate.d/cxfs_client file specifies the manner in which cxfs_client logs are rotated
SGI ProPack supports the CXFS mount scripts. See “CXFS Mount Scripts” in Chapter 1 and the CXFS Administration Guide for SGI InfiniteStorage.
The following sections highlight limitations and considerations for client-only SGI ProPack nodes.
![]() | Note: For limitations and considerations for any SGI ProPack node, see the information in CXFS Administration Guide for SGI InfiniteStorage. |
On systems running SUSE Linux Enterprise Server 10 (SLES 10) that are greater than 64 CPUs, there are issues with using the md driver and CXFS. The md driver holds the BKL (Big Kernel Lock), which is a single, system-wide spin lock. Attempting to acquire this lock can add substantial latency to a driver's operation, which in turn holds off other processes such as CXFS. The delay causes CXFS to lose membership. This problem has been observed specifically when an md pair RAID split is done, such as the following:
raidsetfaulty /dev/md1 /dev/path/to/partition |
The CXFS client-only software will be initially installed and configured by SGI personnel. This section provides an overview of those procedures. You can use the information in this section to verify the installation.
![]() | Note: Package version numbers shown here are examples; your installed system may differ. |
This section covers the following:
Installing the CXFS client software for SGI ProPack requires approximately 50-200 MB of space, depending upon the packages installed at your site.
To install the required software on an SGI ProPack client-only node, SGI personnel will do the following:
Read the SGI InfiniteStorage Software Platform release notes, CXFS general release notes, and CXFS SGI ProPack release notes in the /docs directory on the ISSP DVD and any late-breaking caveats on Supportfolio to learn about any changes in the installation procedure.
Confirm that you have installed SLES 10 SP1. (See the /etc/SuSE-release file.) Ensure that you have the appropriate kernel as listed in the SGI InfiniteStorage Software Platform release notes.
Confirm that you have installed SGI ProPack 5 SP 5. (See the /etc/sgi-release file.) For more information, see the SGI ProPack 5 for Linux Service Pack 5 Start Here.
Install any required patches. See the SGI ProPack releasenotes/README file for more information.
If you had to install software in one of the above steps, reboot the system:
propack# /sbin/reboot |
Transfer the client software that was downloaded onto a server-capable administration node during its installation procedure using ftp, rcp, or scp . The location of the tarball on the server will be as follows:
/usr/cluster/client-dist/CXFS_VERSION/linux/sles10sp1/ARCHITECTURE/cxfs-client.tar.gz |
For example, for an Altix ia64 server:
/usr/cluster/client-dist/5.0.0.3/linux/sles10sp1/ia64/cxfs-client.tar.gz |
Disassemble the downloaded tarball:
propack# tar -zxvf tarball |
Install the CXFS software:
Including GRIOv2:
propack# rpm -Uvh *.rpm |
Without GRIOv2:
propack# rpm -Uvh cxfs*rpm sgi*rpm |
Edit the /etc/cluster/config/cxfs_client.options file as necessary. See the cxfs_client (1M) man page.
Reboot the system with the newly installed kernel:
propack# reboot |
The cxfs_utils package includes a Performance Co-Pilot (PCP) agent for monitoring CXFS heartbeat, CMS status and other statistics. If you want to use this feature, you must also install the following PCP packages:
pcp-open |
pcp-sgi |
These packages and are included on the first and second SGI ProPack CDs (respectively). You can obtain the open source PCP package from ftp://oss.sgi.com/projects/pcp/download
On the SGI ProPack platform, the cxfs_client software automatically detects the world wide port names (WWPNs) of any supported host bus adapters (HBAs) in the system that are connected to a switch that is configured in the cluster database. These HBAs will then be available for fencing.
However, if no WWPNs are detected, there will be messages logged to the /var/log/cxfs_client file.
If no WWPNs are detected, you can manually specify the WWPNs in the /etc/fencing.conf fencing file.
![]() | Note: This method does not work if the WWPNs are partially discovered. |
The /etc/fencing.conf file enumerates the WWPN for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for the HBA WWPN as a 64-bit hexadecimal number.
![]() | Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric. |
If used, /etc/fencing.conf must contain a simple list of WWPNs, one per line. You must update it whenever the HBA configuration changes, including the replacement of an HBA.
Do the following:
Set up the switch and HBA. See the release notes for supported hardware.
Determine the HBA WWPN: Follow the Fibre Channel cable on the back of the node to determine the port to which it is connected in the switch. Ports are numbered beginning with 0. (For example, if there are 8 ports, they will be numbered 0 through 7.)
Use the telnet command to connect to the switch and log in as user admin. (On Brocade switches, the password is password by default).
Execute the switchshow command to display the switches and their WWPN numbers.
For example:
brocade04:admin> switchshow switchName: brocade04 switchType: 2.4 switchState: Online switchRole: Principal switchDomain: 6 switchId: fffc06 switchWwn: 10:00:00:60:69:12:11:9e switchBeacon: OFF port 0: sw Online F-Port 20:00:00:01:73:00:2c:0b port 1: cu Online F-Port 21:00:00:e0:8b:02:36:49 port 2: cu Online F-Port 21:00:00:e0:8b:02:12:49 port 3: sw Online F-Port 20:00:00:01:73:00:2d:3e port 4: cu Online F-Port 21:00:00:e0:8b:02:18:96 port 5: cu Online F-Port 21:00:00:e0:8b:00:90:8e port 6: sw Online F-Port 20:00:00:01:73:00:3b:5f port 7: sw Online F-Port 20:00:00:01:73:00:33:76 port 8: sw Online F-Port 21:00:00:e0:8b:01:d2:57 port 9: sw Online F-Port 21:00:00:e0:8b:01:0c:57 port 10: sw Online F-Port 20:08:00:a0:b8:0c:13:c9 port 11: sw Online F-Port 20:0a:00:a0:b8:0c:04:5a port 12: sw Online F-Port 20:0c:00:a0:b8:0c:24:76 port 13: sw Online L-Port 1 public port 14: sw No_Light port 15: cu Online F-Port 21:00:00:e0:8b:00:42:d8 |
The WWPN is the hexadecimal string to the right of the port number. For example, the WWPN for port 0 is 2000000173002c0b (you must remove the colons from the WWPN reported in the switchshow output to produce the string to be used in the fencing file).
Edit or create /etc/fencing.conf and add the WWPN for the port determined in step 2. (Comment lines begin with #.)
For dual-ported HBAs, you must include the WWPNs of any ports that are used to access cluster disks. This may result in multiple WWPNs per HBA in the file; the numbers will probably differ by a single digit.
For example, if you determined that port 0 is the port connected to the switch, your fencing file should contain the following:
# WWPN of the HBA installed on this system # 2000000173002c0b |
To configure fencing, see the CXFS Administration Guide for SGI InfiniteStorage.
The cxfs_cluster service will be invoked automatically during normal system startup and shutdown procedures. This script starts and stops the cxfs_client daemon.
To start cxfs_client manually, enter the following:
propack# service cxfs_client start Loading cxfs modules: [ OK ] Mounting devfs filesystems: [ OK ] Starting cxfs client: [ OK ] |
To stop CXFS processes manually, enter the following command:
propack# service cxfs_client stop Stopping cxfs client: [ OK ] |
This section discusses the following:
You can modify the CXFS client daemon (/usr/cluster/bin/cxfs_client ) by placing options in the cxfs_client.options file:
/etc/cluster/config/cxfs_client.options |
The available options are documented in the cxfs_client man page.
![]() | Caution: Some of the options are intended to be used internally by SGI only for testing purposes and do not represent supported configurations. Consult your SGI service representative before making any changes. |
For example, to see if cxfs_client is using the options in cxfs_client.options, enter the following:
propack# ps -ax | grep cxfs_client 3612 ? S 0:00 /usr/cluster/bin/cxfs_client -i cxfs3-5 3841 pts/0 S 0:00 grep cxfs_client |
When cxfs_client needs to rescan disk buses, it executes the /var/cluster/cxfs_client-scripts/cxfs-reprobe script. This requires the use of parameters in SGI ProPack due to limitations in the Linux SCSI layer. You can export these parameters from the /etc/cluster/config/cxfs_client.options file.
The script detects the presence of the SCSI and/or XSCSI layers on the system and defaults to probing whichever layers are detected. You can override this decision by setting CXFS_PROBE_SCSI and/or CXFS_PROBE_XSCSI to either 0 (to disable the probe) or 1 (to force the probe) on the appropriate bus.
When an XSCSI scan is performed, all buses are scanned by default. You can override this decision by specifying a space-separated list of buses in CXFS_PROBE_XSCSI_BUSES. (If you include space, you must enclose the list within single quotation marks.) For example, for SGI ProPack 4:
export CXFS_PROBE_XSCSI_BUSES='/dev/xscsi/pci01.03.0-1/bus /dev/xscsi/pci02.01.0-2/bus' |
For SGI ProPack 5:
export CXFS_PROBE_XSCSI_BUSES='/dev/xscsi/pci0001:00:03.0-1/bus /dev/xscsi/pci0002:00:01.0-2/bus' |
When a SCSI scan is performed, a fixed range of buses/channels/IDs and LUNs are scanned; these ranges may need to be changed to ensure that all devices are found. The ranges can also be reduced to increase scanning speed if a smaller space is sufficient.
The following summarizes the environment variables (separate multiple values by white space and enclose withing single quotation marks):
CXFS_PROBE_SCSI=0|1 | |
Stops (0) or forces (1) a SCSI probe. Default: 1 if SCSI | |
CXFS_PROBE_SCSI_BUSES=BusList | |
Scans the buses listed. Default: 0 1 2 | |
CXFS_PROBE_SCSI_CHANNELS=ChannelList | |
Scans the channels listed. Default: 0 | |
CXFS_PROBE_SCSI_IDS=IDList | |
Scans the IDS listed. Default: 0 1 2 3 | |
CXFS_PROBE_SCSI_LUNS=LunList | |
Scans the LUNs listed. Default: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | |
CXFS_PROBE_XSCSI=0|1 | |
Stops (1) or forces (1) an XSCSI probe. Default: 1 if XSCSI | |
CXFS_PROBE_XSCSI_BUSES=BusList | |
Scans the buses listed. Default: all XSCSI buses |
For example, the following would only scan the first two SCSI buses:
export CXFS_PROBE_SCSI_BUSES='0 1' |
The following would scan 16 LUNs on each bus, channel, and ID combination (all on one line):
export CXFS_PROBE_SCSI_LUNS='0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15' |
Other options within the /etc/cluster/config/cxfs_client.options file begin with a - character. Following is an example cxfs_client.options file:
propack# Example cxfs_client.options file # -Dnormal -serror export CXFS_PROBE_SCSI_BUSSES=1 export CXFS_PROBE_SCSI_LUNS='0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20' |
![]() | Note: The - character or the term export must start in the first position of each line in the cxfs_client.options file; otherwise, they are ignored by the cxfs_client service. |
CXFS supports guaranteed-rate I/O (GRIO) version 2 on the SGI ProPack client-only platform.
GRIO is disabled by default on SGI ProPack. To enable GRIO, change the following line in /etc/cluster/config/cxfs_client.options from:
export GRIO2=off |
to:
export GRIO2=on |
Application bandwidth reservations must be explicitly released by the application before exit. If the application terminates unexpectedly or is killed, its bandwidth reservations are not automatically released and will cause a bandwidth leak. If this happens, the lost bandwidth could be recovered by rebooting the client node.
An SGI ProPack client-only node can mount a GRIO-managed filesystem and supports application- and node-level reservations. An SGI ProPack client-only node client will interoperate with the dynamic bandwidth allocator for all I/O outside of any reservation.
For more information, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and the Guaranteed-Rate I/O Version 2 Guide.
Following is an example of the /etc/failover2.conf file on SGI ProPack:
/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port1/lun4/disc, affinity=1 /dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port2/lun4/disc, affinity=2 /dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port1/lun4/disc, affinity=1 /dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port2/lun4/disc, affinity=2 /dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port1/lun4/disc, affinity=4 /dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port2/lun4/disc, affinity=3 preferred /dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port1/lun4/disc, affinity=4 /dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port2/lun4/disc, affinity=3 |
For more information, see “XVM Failover and CXFS” in Chapter 1, the comments in the /etc/failover2.conf.example file, CXFS Administration Guide for SGI InfiniteStorage, and the XVM Volume Manager Administrator's Guide.
Retain the following information for SGI ProPack nodes:
propack# uname -a |
The CXFS packages you are running:
propack# rpm -q cxfs_client sgi-cxfs-kmp cxfs_utils cxfs-xvm-cmds |
The number and types of processors in your machine:
propack# cat /proc/cpuinfo |
The hardware installed on your machine:
propack# /sbin/lspci |
Modules that are loaded on your machine:
propack# /sbin/lsmod |
The /var/log/cxfs_client log file
Any messages that appeared in the system logs immediately before the system exhibited the problem.
Output about the cluster obtained from the cxfsdump utility run on a server-capable administration node.
After a system kernel panic, the debugger information from the KDB built-in kernel debugger.
Fibre Channel HBA World Wide name mapping:
cat /sys/class/fc_transport/bus_ID/node_name |
For example:
cat /sys/class/fc_transport/11:0:0:0/node_name |
The bus_ID value is the output of hwinfo --disk in the SysFS BusID field.
Output from the following commands:
Information from the following files:
/var/log/messages /var/log/cxfs_client (for client-only nodes) /etc/failover.conf (for XVM failover version 1) /etc/failover2.conf (for XVM failover version 2) /etc/hosts /proc/discontig |
Output from the following commands:
/usr/cluster/bin/cdbutil gettree '#' /usr/bin/hinv /usr/bin/topology /sbin/xvm show -v phys /sbin/xvm show -top -v vol /bin/netstat -ia |