When you install the CXFS software, there are some system file considerations you must take into account. The network configuration is critical. Each node in the cluster must be able to communicate with every other node in the cluster by both logical name and IP address without going through any other network routing; proper name resolution is key. SGI recommends static routing.
This section discusses the following:
After completing the steps discussed in this chapter, see Chapter 9, “Initial Setup of the Cluster”. For information about upgrades, see “CXFS Release Versions and Rolling Upgrades” in Chapter 12.
The optional /etc/exports file on each node describes the filesystems that are being exported to NFS clients. For optimal performance, use the mountpoint ( mp) option for CXFS filesystems in /etc/exports that are exported with NFS. For more information, see the Linux exports(5) man page.
If the /etc/exports file contains a CXFS mount point, then when the system is booted NFS will export the empty mount point because the exports are done before CXFS is running. When CXFS on the node joins membership and starts mounting filesystems, the clconfd-pre-mount script searches the /etc/exports file looking for the mountpoint that is being mounted. If found, the script unexports the mountpoint directory because if it did not the CXFS mount would fail. After successfully mounting the filesystem, the clconfd-post-mount script will search the /etc/exports file and export the mount point if it is found in the /etc/exports file. For more information, see “CXFS Mount Scripts” in Chapter 12.
This section discusses system files on administration nodes:
The /etc/services file on each CXFS administration contains entries for sgi-cad and sgi-crsd. The port numbers assigned for these processes must be the same in all nodes in the pool.
The following shows an example of /etc/services entries for sgi-cad and sgi-crsd:
sgi-crsd 7500/udp # Cluster reset services daemon sgi-cad 5435/tcp # Cluster Admin daemon |
The cad.options file on each server-capable administration node contains the list of parameters that the cluster administration daemon reads when the cad process (which provides cluster information) is started. The file is located as follows:
/etc/cluster/config/cad.options |
You can set the following options in the cad.options file:
| --append_log | Appends cad logging information to the cad log file instead of overwriting it. |
| --log_file file | Specifies the cad log filename. Alternately, this can be specified as -lf file. |
| --tcp_port | Specifies a TCP port number to be used by cad . The default is port 5435. |
| -vvvv | Specifies the verbosity level. The number of v characters indicates the level of logging. Setting -v logs the fewest messages; setting -vvvv logs the highest number of messages. |
The default file has the following options:
-lf /var/log/cxfs/cad_log --append_log |
The following example shows an /etc/config/cad.options file that uses a medium-level of verbosity:
-vv -lf /var/log/cxfs/cad_log --append_log |
The default log file is /var/log/cxfs/cad_log. Error and warning messages are appended to the log file if log file is already present.
The contents of the /etc/config/cad.options file cannot be modified using cxfs_admin or the GUI.
If you make a change to the cad.options file at any time other than initial configuration, you must restart the cad processes in order for these changes to take effect. You can do this by rebooting the nodes or by entering the following command:
server-admin# service cxfs_cluster restart |
If you execute this command on a running cluster, it will remain up and running. However, the GUI will lose connection with the cad daemon; the GUI will prompt you to reconnect.
The fs2d.options file on each server-capable administration node contains the list of parameters that the fs2d daemon reads when the process is started. (The fs2d daemon manages the distribution of the cluster database (CDB) across the server-capable administration nodes in the pool.) The file is located as follows:
/etc/cluster/config/fs2d.options |
Table 8-1 shows the options can that can be set in the fs2d.options file.
Table 8-1. fs2d.options File Options
If you use the default values for these options, the system will be configured so that all log messages of level info or less, and all trace messages for transaction events, are sent to the /var/log/cxfs/fs2d_log file. When the file size reaches 10 MB, this file will be moved to its namesake with the .old extension and logging will roll over to a new file of the same name. A single message will not be split across files.
If you make a change to the fs2d.options file at any time other than the initial configuration time, you must restart the fs2d processes in order for those changes to take effect. You can do this by rebooting the server-capable administration nodes or by entering the following command:
server-admin# service cxfs_cluster restart |
If you execute this command on a running cluster, it should remain up and running. However, the GUI will lose connection with the cad daemon; the GUI will prompt you to reconnect.
The following example shows an /etc/config/fs2d.options file that directs logging and tracing information as follows:
All log events are sent to: /var/log/messages
Tracing information for RPCs, updates, and transactions are sent to /var/log/cxfs/fs2d_ops1.
When the size of this file exceeds 100,000,000 bytes, this file is renamed to /var/log/cxfs/fs2d_ops1.old and a new file /var/log/cxfs/fs2d_ops1 is created. A single message is not split across files.
(Line breaks added for readability.)
-logevents all -loglevel trace -logdest syslog -trace rpcs -trace updates -trace transactions -tracefile /var/log/cxfs/fs2d_ops1 -tracefilemax 100000000 |
The following example shows an /etc/config/fs2d.options file that directs all log and trace messages into one file, /var/log/cxfs/fs2d_chaos6, for which a maximum size of 100,000,000 bytes is specified. -tracelog directs the tracing to the log file.
(Line breaks added for readability.)
-logevents all -loglevel trace -trace rpcs -trace updates -trace transactions -tracelog -logfile /var/log/cxfs/fs2d_chaos6 -logfilemax 100000000 -logdest logfile. |
You can use the clconfd.options file on each server-capable administration node to contain a list of nondefault parameters that the clconfd daemon will read when the process is started. To use this feature, create the following file:
/etc/cluster/config/clconfd.options |
Table 8-2 shows the options that can be set in the fs2d.options file.
Table 8-2. clconfd.options File Options
Option | Description |
|---|---|
-c CDBfile | Reads the cluster database configuration from the specified CDBfile file. The default file is /var/cluster/cdb/cdb.db. |
-d debugfile | Enables printing debug information to the specified file debugfile. The default is to print no information. |
-h | Prints a help message for clconfd.options. |
-l | Runs clconfd in the foreground. (For SGI development debugging purposes only. Do not use this option unless directed to do so by SGI support.) The default is to run clconfd in the background. |
-s loglevel | Specifies the log level to use for logging to standard error. The default is 0 (no logging). For information about log levels, see “Configure Log Groups with the GUI” in Chapter 10. |
-R | Disables real-time scheduling. By default, real-time scheduling is enabled. |
For example, to print hafence debug information to the file /tmp/hafence.log, add the following line to the clconfd.options file:
-d /tmp/hafence.log |
If you make a change to the clconfd.options file at any time other than the initial configuration time, you must restart the clconfd processes in order for those changes to take effect. You can do this by rebooting the server-capable administration nodes or by entering the following command:
server-admin# service cxfs restart |
This section discusses enabling and disabling GRIOv2 the following for server-capable administration nodes:
 | Note: GRIOv2 requires a license key on the server-capable administration nodes. |
To enable GRIOv2 on a server-capable administration node, do the following:
Turn the grio2 flag on with the chkconfig(8) command:
server# chkconfig grio2 on
Reboot the system.
This section discusses the following:
XVM path failover creates an infrastructure for the definition and management of multiple paths to a single disk device or logical unit (LUN). XVM selects a path for I/O from the host through the fabric to the RAID, using a particular HBA at the host end and a particular controller at the RAID end. XVM can be controlled to select paths that result in the best I/O performance. The key choices are:
RAID controller selection
HBA selection
Unnecessary switching between RAID controllers to access a LUN can degrade performance considerably. If more selected paths go through one HBA than through another, resource contention in the first HBA will slow down I/O while the other one is underused.
To avoid these problems, XVM uses path failover and the path manager feature. You must specify appropriate controller groupings, preferred controller paths, and/or HBA usage in the /etc/failover2.conf file on each node in the CXFS cluster as needed.
| Note: SGI requires SGIAVT or ALUA mode. See “Changing SGIRDAC Mode to SGIAVT Mode for SGI RAID” in Appendix F. |
To create the failover2.conf file for each node in the cluster, do the following:
Create the preliminary file on the active CXFS metadata server:
mds# mk_failover2.pl -C preliminary
Edit the preliminary file to make any desired affinity changes.
Copy the edited preliminary file to each of the other Linux server-capable administration nodes, Linux client-only nodes, and Mac OS client-only nodes.
Using the edited preliminary file as a base, create a node-specific file on each node and rename it:
Manually configure the failover2.conf file for Windows, as directed in CXFS 7 Client-Only Guide for SGI InfiniteStorage.
Also see:
The example file installed in /etc/failover2.conf.example
The comments in the failover2.conf file
| Note: The xvm command is provided on all CXFS platforms. You can use xvm on a client-only node to display configuration information. To make administrative changes, you must run xvm as root on a server-capable administration node. |
The following are useful xvm commands related to XVM failover:
Get information about a command:
# xvm help -verbose foconfig # xvm help -verbose foswitch # xvm help -verbose show
Display information about physical volumes ( physvols):
# xvm show -verbose physvol # xvm show -verbose physvol | fgrep affinity > templatefile
Push the information in the /etc/failover2.conf file to the kernel:
# xvm foconfig -init
Switch paths for physical volumes:
# xvm foswitch -cluster -preferred phys # xvm foswitch -cluster -preferred phys/physvolname # xvm foswitch -cluster -affinity affinityvalue phys # xvm foswitch -cluster -setaffinity affinityvalue phys/physvolname
Show current paths for physical volumes not on preferred paths:
# xvm show -verbose physvol | fgrep current | fgrep -v preferred
Switch to the specified device number for the local node:
# xvm foswitch -dev newdevice
For details, see the XVM Volume Manager Administrator Guide and the xvm(8) man page.