This section contains the following information about CXFS on Linux systems:

This section provides an overview of the Fibre Channel host bus adapter (HBA) installation information for Linux nodes.

The installation may be performed by you or by a qualified service representative for your hardware. See the Linux operating system documentation and the documentation for your hardware platform.

The driver requirements are as follows:

You must ensure that the HBA driver is loaded prior to CXFS initialization by building the module into the initial RAM disk automatically or manually. For example, using the QLogic card and the qla2200 driver:

You should then verify the appropriate initrd information:

The system must be rebooted (and when using LILO, LILO must be rerun) for the new initrd image to take effect.

Instead of this procedure, you could also modify the /etc/rc.sysinit script to load the qla2200 driver early in the initscript sequence.

This section provides an overview of the steps that you will perform on your Linux nodes prior to installing the CXFS software. It contains the following sections:

ping nodeIPaddress

linux# ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) from 128.162.240.141 : 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.310 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.122 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.127 ms

The CXFS software will be initially installed and configured by SGI personnel. This section provides an overview of those procedures. You can use the information in this section to verify the installation.

Table 5-1 and Table 5-2 provide examples of the differences in package extensions among the various processor classes supported by CXFS.

---

Note: The kernel package extensions vary by architecture. Ensure that you install the appropriate package for your processor architecture.

---

Linux Installation Procedure

---

Note: Specific packages listed here are examples and may not match the released product.

---

Installing the CXFS client software for Linux requires approximately 50-200 MB of space, depending upon the packages installed at your site.

To install the required software on a Linux node, SGI personnel will do the following:

1. Read the SGI InfiniteStorage Software Platform release notes, CXFS general release notes, and CXFS Linux release notes in the /docs directory on the ISSP DVD and any late-breaking caveats on Supportfolio.

2. Verify that the node is running a supported Linux distribution, according to the CXFS for Linux release notes. See the Red Hat /etc/redhat-release or SLES /etc/SuSE-release files.

3. Transfer the client software that was downloaded onto a server-capable administration node during its installation procedure using ftp, rcp, or scp . The location of the tarball on the server will be as follows:

   /usr/cluster/client-dist/CXFS_VERSION/linux/LINUX_VERSION/ARCHITECTURE/cxfs-client.tar.gz

   
   

   For example, for SLES 10 SP 1 running on an x86_64 architecture machine:

   /usr/cluster/client-dist/5.0.0.3/linux/sles10sp1/x8664/cxfs-client.tar.gz

   
   

4. Disassemble the downloaded tarball:

   linux# tar -zxvf tarball

   
   

5. Install the CXFS software:

   • For RHEL:

     • Including GRIOv2:

       rhel# rpm -Uvh *.rpm

       
       

     • Without GRIOv2:

       rhel# rpm -Uvh cxfs*rpm kernel*rpm sgi*rpm

       
       

   • For SLES:

     • Including GRIOv2:

       sles# rpm -Uvh *.rpm

       
       

     • Without GRIOv2:

       sles# rpm -Uvh cxfs*rpm sgi*rpm

       
       

6. Edit the /etc/cluster/config/cxfs_client.options file as necessary. See the “Maintenance for Linux” and the cxfs_client(1M) man page.

7. Reboot the system with the newly installed kernel:

   linux# reboot

   
   

I/O fencing is required on Linux nodes in order to protect data integrity of the filesystems in the cluster. The cxfs_client software automatically detects the world wide port names (WWPNs) of any supported host bus adapters (HBAs) for Linux nodes that are connected to a switch that is configured in the cluster database. These HBAs are available for fencing.

However, if no WWPNs are detected, there will be messages about loading the HBA/SNIA library logged to the /var/log/cxfs_client file.

If no WWPNs are detected, you can manually specify the WWPNs in the fencing file.

---

Note: This method does not work if the WWPNs are partially discovered.

---

The /etc/fencing.conf file enumerates the WWPNs for all of the HBAs that will be used to mount a CXFS filesystem. There must be a line for each HBA WWPN as a 64-bit hexadecimal number.

---

Note: The WWPN is that of the HBA itself, not any of the devices that are visible to that HBA in the fabric.

---

If used, /etc/fencing.conf must contain a simple list of WWPNs, one per line. You must update it whenever the HBA configuration changes, including the replacement of an HBA.

Do the following:

The cxfs_client service will be invoked automatically during normal system startup and shutdown procedures. This script starts and stops the cxfs_client daemon.

To start up cxfs_client manually, enter the following:

linux# service cxfs_client start
Loading cxfs modules:                                      [  OK  ]
Mounting devfs filesystems:                                [  OK  ]
Starting cxfs client:                                      [  OK  ]

To stop cxfs_client manually, enter the following:

linux# service cxfs_client stop
Stopping cxfs client:                                      [  OK  ]

To stop and then start cxfs_client manually, enter the following:

linux# service cxfs_client restart
Stopping cxfs client:                                      [  OK  ]

To see the current status, use the status argument. For example:

linux# service cxfs_client status
cxfs_client status [timestamp Apr 20 14:54:30 / generation 4364]

CXFS client:
    state: stable (5), cms: up, xvm: up, fs: up
Cluster:
    connies_cluster (707) - enabled
Local:
    ceara (7) - enabled
Nodes:
    aiden      enabled  up    12    
    brenna     enabled  DOWN  10    
    brigid     enabled  up    11    
    ceara      enabled  up    7     
    chili      enabled  up    4     
    cxfsibm2   enabled  up    9     
    cxfssun4   enabled  up    5     
    daghada    enabled  up    8     
    flynn      enabled  up    2     
    gaeth      enabled  up    0     
    minnesota  enabled  up    6     
    rowan      enabled  up    3     
    rylie      enabled  up    1     
Filesystems:
    concatfs   enabled  mounted          concatfs             /concatfs
    stripefs   enabled  mounted          stripefs             /stripefs
    tp9300_stripefs enabled  forced mounted   tp9300_stripefs      /tp9300_stripefs
cxfs_client is running.

For example, if cxfs_client is stopped:

linux# service cxfs_client status
cxfs_client is stopped

This section contains information about maintenance procedures for CXFS on Linux.

Modifying the CXFS Software for Linux

You can modify the behavior of the CXFS client daemon ( cxfs_client) by placing options in the /etc/cluster/config/cxfs_client.options file. The available options are documented in the cxfs_client man page.

---

Caution: Some of the options are intended to be used internally by SGI only for testing purposes and do not represent supported configurations. Consult your SGI service representative before making any changes.

---

To see if cxfs_client is using the options in cxfs_client.options, enter the following:

linux# ps -ax | grep cxfs_client 3612 ? S 0:00 /usr/cluster/bin/cxfs_client -i cxfs3-5 3841 pts/0 S 0:00 grep cxfs_client

/var/cluster/cxfs_client-scripts/cxfs-reprobe

When cxfs_client needs to rescan disk buses, it executes the /var/cluster/cxfs_client-scripts/cxfs-reprobe script. This requires the use of parameters in RHEL due to limitations in the SCSI layer. You can export these parameters from the /etc/cluster/config/cxfs_client.options file.

The script detects the presence of the SCSI and/or XSCSI layers on the system and defaults to probing whichever layers are detected. You can override this decision by setting CXFS_PROBE_SCSI (for Linux SCSI) or CXFS_PROBE_XSCSI (for Linux XSCSI) to either 0 (to disable the probe) or 1 (to force the probe).

When an XSCSI scan is performed, all buses are scanned by default. You can override this by specifying a space-separated list of buses in CXFS_PROBE_XSCSI_BUSES. (If you include space, you must enclose the list within single quotation marks.) For example:

export CXFS_PROBE_XSCSI_BUSES='/dev/xscsi/pci01.03.0-1/bus /dev/xscsi/pci02.01.0-2/bus'

When a SCSI scan is performed, a fixed range of buses/channels/IDs and LUNs are scanned; these ranges may need to be changed to ensure that all devices are found. The ranges can also be reduced to increase scanning speed if a smaller space is sufficient.

The following summarizes the environment variables (separate multiple values by white space and enclose withing single quotation marks):

For example, the following would only scan the first two SCSI buses:

export CXFS_PROBE_SCSI_BUSES='0 1'

The following would scan 16 LUNs on each bus, channel, and ID combination (all on one line):

export CXFS_PROBE_SCSI_LUNS='0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15'

Other options within the /etc/cluster/config/cxfs_client.options file begin with a - character. Following is an example cxfs_client.options file:

# Example cxfs_client.options file
#
-Dnormal -serror
export CXFS_PROBE_SCSI_BUSSES=1
export CXFS_PROBE_SCSI_LUNS='0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20'

---

Note: The - character or the term export must start in the first position of each line in the cxfs_client.options file; otherwise, they are ignored by the cxfs_client service.

---

CXFS supports guaranteed-rate I/O (GRIO) version 2 on the Linux platform. However, GRIO is disabled by default on Linux. To enable GRIO, change the following line in /etc/cluster/config/cxfs_client.options from:

export GRIO2=off

to:

export GRIO2=on

Application bandwidth reservations must be explicitly released by the application before exit. If the application terminates unexpectedly or is killed, its bandwidth reservations are not automatically released and will cause a bandwidth leak. If this happens, the lost bandwidth could be recovered by rebooting the client node.

A Linux client can mount a GRIO-managed filesystem and supports application- and node-level reservations. A Linux client will interoperate with the dynamic bandwidth allocator for all I/O outside of any reservation.

For more information, see “Guaranteed-Rate I/O (GRIO) and CXFS” in Chapter 1 and the Guaranteed-Rate I/O Version 2 Guide.

Following is an example of the /etc/failover2.conf file on a Linux system:

/dev/disk/by-path/pci-0000:06:02.1-fc-0x200800a0b8184c8e:0x0000000000000000 affinity=0 preferred
/dev/disk/by-path/pci-0000:06:02.1-fc-0x200900a0b8184c8d:0x0000000000000000 affinity=1

For more information, see:

This section discusses the following:

For general troubleshooting information, see Chapter 11, “General Troubleshooting” and Appendix D, “Error Messages”.

linux# ps -ax | grep cxfs_client

xfs                       0:off  1:off  2:off  3:off  4:off  5:off  6:off

When reporting a problem about a Linux node to SGI, you should retain the following information: