You will use one of the following commands for CXFS administration:

Figure 12-1 shows the order in which CXFS programs take their configuration options.

Figure 12-1. Precedence of Configuration Options

SGI lets you upgrade of a subset of nodes from X.anything to X.anything within the same major-release thread (X). This policy lets you to keep your cluster running and filesystems available during the temporary upgrade process.

To identify compatible CXFS releases, see the CXFS MultiOS Software Compatibility Matrix that is posted on Supportfolio:

https://support.sgi.com/content_request/139840/index.html

After the upgrade process is complete, all nodes should be running the same major-level release X.Y (such as 4.0), or any minor-level release with the same major level X.Y.anything (such as 4.0.3).

---

Caution: You must upgrade all server-capable administration nodes before upgrading any client-only nodes (server-capable administration nodes must run the same or later release as client-only nodes.) Operating a cluster with clients running a mixture of older and newer CXFS versions will result in a performance loss. Relocation to a server-capable administration node that is running an older CXFS version is not supported. Although clients that are not upgraded might continue to function in the CXFS cluster without problems, new CXFS functionality may not be enabled until all clients are upgraded; SGI does not provide support for any CXFS problems encountered on the clients that are not upgraded.

---

Using the 3.4.2 release as an example, a production cluster could contain server-capable administration nodes running 3.4.2 and client-only nodes running 3.4, 3.4.1, and 3.4.2; it could contain client-only nodes running 3.4.3 only because there was no server platforms included in 3.4.3. It should not contain any nodes running 3.3.

This section discusses the following:

Example Upgrade Process

The following figures show an example upgrade procedure for a cluster with three server-capable administration nodes and two filesystems ( /fs1 and /fs2), in which all nodes are running CXFS 4.0 at the beginning and Node2 is the standby node.

Figure 12-2. Example Rolling Upgrade Procedure (part 1)

Figure 12-3. Example Rolling Upgrade Procedure (part 2)

Figure 12-4. Example Rolling Upgrade Procedure (part 3)

---

Note: Relocation is disabled by default.

---

To use relocation, you must enable relocation on the active metadata server. To enable relocation, reset the cxfs_relocation_ok parameter as follows:

To disable relocation, do the following:

See also “Relocation” in Chapter 1.

Table 12-1 summarizes the /etc/init.d initialization commands used for the CXFS control daemon and the cluster administration daemons. For more information about commands, see “CXFS Tools” in Chapter 1.

To add or modify a switch, run the following commands on a server-capable administration node:

server-admin# /usr/cluster/bin/hafence -a -s switchname -u username -p password -m mask [-L vendorname]

To raise the fence for a node:

server-admin# /usr/cluster/bin/hafence -r nodename

To lower the fence for a node:

server-admin# /usr/cluster/bin/hafence -l nodename

To query switch status:

server-admin# /usr/cluster/bin/hafence -q -s switchname

Usage notes:

For example, the following defines a QLogic switch named myqlswitch and uses no masking:

server-admin# /usr/cluster/bin/hafence -a -s myqlswitch -u admin -p *** -L qlogic

---

Note: Vendor plugin libraries should be installed in a directory that is in the platform-specific search path of the dynamic linker, typically the same location as the fencing library, libcrf.so. The above command line will attempt to load the libcrf_qlogic.so library.

---

The following masks port numbers 2 and 3:

server-admin# /usr/cluster/bin/hafence -a -s myqlswitch -u admin -p *** -m "#2,3" -L qlogic

The following lowers the fence for client1:

server-admin# /usr/cluster/bin/hafence -l client1

The following raises the fence for client1:

server-admin# /usr/cluster/bin/hafence -r client1

The following queries port status for all switches defined in the cluster database:

server-admin# /usr/cluster/bin/hafence -q

For more information, see the hafence(1M) man page. See the release notes for supported switches.

CXFS uses the following ports:

For more information, see Appendix B, “IP Filtering for the CXFS Private Network”.

Table 12-2 summarizes the CXFS chkconfig arguments for server-capable administration nodes. These settings can be modified by the CXFS GUI or by the administrator. These settings only control the processes, not the cluster. Stopping the processes that control the cluster will not stop the cluster (that is, will not drop the cluster membership or lose access to CXFS filesystems and cluster volumes), and starting the processes will start the cluster only if the CXFS services are marked as activated in the database.

The following shows the settings of the arguments on server-capable administration nodes:

server-admin# chkconfig --list | grep cxfs fam
cxfs_cluster    0:off   1:off   2:on    3:on    4:on    5:on    6:off
cxfs            0:off   1:off   2:on    3:on    4:on    5:on    6:off
fam             0:off   1:off   2:on    3:on    4:on    5:on    6:off

Table 12-2. chkconfig Arguments for Server-Capable Administration Nodes

Argument	Description
cxfs_cluster	Controls the cluster administration daemons (fs2d, crsd, cad , and cmond). If this argument is off, the database daemons will not be started at the next reboot and the local copy of the database will not be updated if you make changes to the cluster configuration on the other nodes. This could cause problems later, especially if a majority of nodes are not running the database daemons. If the database daemons are not running, the cluster database will not be accessible locally and the node will not be configured to join the cluster.
cxfs	Controls the clconfd daemon and whether or not the cxfs_shutdown command is used during a system shutdown. The cxfs_shutdown command attempts to withdraw from the cluster gracefully before rebooting. Otherwise, the reboot is seen as a failure and the other nodes must recover from it. Note: clconfd cannot start unless fs2d is already running.
fam	Starts the file alteration monitoring (fam) service, which is required to use the CXFS GUI on Linux nodes.

The CXFS GUI lets you grant or revoke access to a specific GUI task for one or more specific users. By default, only root may execute tasks in the GUI. Access to the task is only allowed on the node to which the GUI is connected; if you want to allow access on another node in the pool, you must connect the GUI to that node and grant access again.

---

Note: You cannot grant or revoke tasks for users with a user ID of 0.

---

GUI tasks operate by executing underlying privileged commands that are normally accessible only to root. When granting access to a task, you are in effect granting access to all of its required underlying commands, which results in also granting access to the other GUI tasks that use the same underlying commands. The cxfs_admin command provides similar functionality with the allow|deny subcommands.

For instructions about granting or revoking GUI privileges, see “Privileges Tasks with the GUI” in Chapter 10.

To see which tasks a specific user can currently access, select View: Users. Select a specific user to see details about the tasks available to that user.

To see which users can currently access a specific task, select View: Task Privileges. Select a specific task to see details about the users who can access it and the privileged commands it requires.

You should install a node as a server-capable administration node only if you intend to use it as a potential metadata server. All other nodes should be installed as client-only nodes. See “Make Most Nodes Client-Only” in Chapter 2.

To transform a server-capable administration node into a client-only node, do the following:

For more information about these tasks, see:

On server-capable administration nodes, the following scripts are provided for execution by the clconfd daemon prior to and after a CXFS filesystem is mounted or unmounted:

The following script is run when needed to reprobe the Fibre Channel controllers on server-capable administration nodes:

/var/cluster/clconfd-scripts/cxfs-reprobe

You can customize these scripts to suit a particular environment. For example, an application could be started when a CXFS filesystem is mounted by extending the cxfs-post-mount script. The application could be terminated by changing the cxfs-pre-umount script.

On server-capable administration nodes, these scripts also allow you to use NFS to export the CXFS filesystems listed in /etc/exports if they are successfully mounted.

The appropriate daemon executes these scripts before and after mounting or unmounting CXFS filesystems specified in the /etc/exports file. The files must be named exactly as above and must have root execute permission.

---

Note: The /etc/exports file describes the filesystems that are being exported to NFS clients. If a CXFS mount point is included in the exports file, the empty mount point is exported unless the filesystem is re-exported after the CXFS mount using the cxfs-post-mount script.

---

The following arguments are passed to the files:

Because the filesystem device name is passed to the scripts, you can write the scripts so that they take different actions for different filesystems; because the exit codes are passed to the -post- files, you can write the scripts to take different actions based on success or failure of the operation.

The clconfd or cxfs_client daemon checks the exit code for these scripts. In the case of failure (nonzero), the following occurs:

This implies that if you do not want a filesystem to be mounted on a host, the cxfs-pre-mount script should return a failure for that filesystem while the cxfs-post-mount script returns success.

---

Note: After the filesystem is unmounted, the mount point is removed.

---

For information about the mount scripts on client-only nodes, see the CXFS 5 Client-Only Guide for SGI InfiniteStorage.

DMF must make all of its DMAPI interface calls through the CXFS active metadata server. The CXFS client nodes do not provide a DMAPI interface to CXFS mounted filesystems. A CXFS client routes all of its communication to DMF through the metadata server. This generally requires that DMF run on the CXFS metadata server. If DMF is managing a CXFS filesystem, DMF will ensure that the filesystem's CXFS metadata server is the DMF server and will use metadata server relocation if necessary to achieve that configuration.

---

Note: DMF data mover processes must only run on the DMF server node and any parallel data mover nodes. Do not run data mover processes on CXFS standby metadata server nodes.

---

DMF requires independent paths to tape drives so that they are not fenced by CXFS. The ports for the tape drive paths on the switch should be masked from fencing in a CXFS configuration.

The SAN must be zoned so that XVM does not failover CXFS filesystem I/O to the paths visible through the tape HBA ports when Fibre Channel port fencing occurs. Therefore, either independent switches or independent switch zones should be used for CXFS/XVM volume paths and DMF tape drive paths.

To use DMF with CXFS, do the following:

To avoid token thrashing between the CXFS client and server, which can result in poor I/O transfer rates, consider setting the DMF configuration file parameter RECALL_NOTIFICATION_RATE to 0 if you are using the Parallel Data Mover Option; see the “Best Practices” chapter in the DMF administrator's guide for details.

For more information about DMF, see the DMF 4 Administrator's Guide for SGI InfiniteStorage.

This section discusses how to discover the active metadata server using various tools:

CXFS GUI and the Active Metadata Server

To discover the active metadata server for a filesystem by using the GUI, do the following:

1. Select View: Filesystems

2. In the view area, click the name of the filesystem you wish to view. The name of the active metadata server is displayed in the details area to the right.

Figure 12-5 shows an example.

Figure 12-5. GUI Window Showing the Metadata Server

cxfs6 # clconf_info

Event at [2004-04-16 09:20:59]

Membership since Fri Apr 16 09:20:56 2004

____________ ______ ________ ______ ______
Node         NodeID Status   Age    CellID
____________ ______ ________ ______ ______
cxfs6             6 up       0           2
cxfs7             7 up       0           1
cxfs8             8 up       0           0
____________ ______ ________ ______ ______
1 CXFS FileSystems
/dev/cxvm/concat0 on /concat0  enabled  server=(cxfs7)  2 client(s)=(cxfs8,cxfs6)

This section tells you how to perform the following:

For more information about states, Chapter 14, “Monitoring Status”. If there are problems, see Chapter 15, “Troubleshooting”.

Cluster Database Shutdown

A cluster database shutdown terminates the following user-space daemons that manage the cluster database:

cad

clconfd

cmond

crsd

fs2d

After shutting down the database on a node, access to the shared filesystems remains available and the node is still a member of the cluster, but the node is not available for database updates. Rebooting of the node results in a restart of all services (restarting the daemons, joining cluster membership, enabling cluster volumes, and mounting CXFS filesystems).

To perform a cluster database shutdown, enter the following on a server-capable administration node:

server-admin# killall -TERM clconfd server-admin# service cxfs_cluster stop

If you also want to disable the daemons from restarting at boot time, enter the following:

server-admin# /sbin/chkconfig grio2 off (If running GRIOv2) server-admin# /sbin/chkconfig cxfs off server-admin# /sbin/chkconfig cxfs_cluster off

For more information, see “chkconfig Arguments”.

Node Status and Cluster Database Shutdown

A cluster database shutdown is appropriate when you want to perform a maintenance operation on the node and then reboot it, returning it to ACTIVE status (as displayed by the GUI) or stable status (as displayed by cxfs_admin).

If you perform a cluster database shutdown, the node status will be DOWN in the GUI or inactive in cxfs_admin, which has the following impacts:

• The node is still considered part of the cluster, but unavailable.

• The node does not get cluster database updates; however, it will be notified of all updates after it is rebooted.

  Missing cluster database updates can cause problems if the kernel portion of CXFS is active. That is, if the node continues to have access to CXFS, the node's kernel level will not see the updates and will not respond to attempts by the remaining nodes to propagate these updates at the kernel level. This in turn will prevent the cluster from acting upon the configuration updates.

---

Note: If the cluster database is shut down on more than half of the server-capable administration nodes, changes cannot be made to the cluster database.

---

Restart the Cluster Database

To restart the cluster database, enter the following:

server-admin# service cxfs_cluster start server-admin# service cxfs start

Forced CXFS Shutdown: Revoke Membership of Local Node

A forced CXFS shutdown (or administrative CXFS stop) is appropriate when you want to shutdown the local node even though it may drop the cluster below its CXFS kernel membership quorum requirement.

CXFS does the following:

• Shuts down all CXFS filesystems on the local node. Any attempts to access the CXFS filesystems will result in an I/O error (you may need to manually unmount the filesystems).

• Removes this node from the CXFS kernel membership.

• Marks the node as DOWN in the GUI or inactive in cxfs_admin.

• Disables access from the local node to cluster-owned XVM volumes.

• Treats the stopped node as a failed node and executes the fail policy defined for the node in the cluster database. See “Fail Policies” in Chapter 2.

---

Caution: A forced CXFS shutdown may cause the cluster to fail if the cluster drops below CXFS kernel membership quorum.

---

If you do a forced CXFS shutdown on an active metadata server, it loses membership immediately. At this point, another potential metadata server must take over (and recover the filesystems) or quorum is lost and a forced CXFS shutdown follows on all nodes.

If you do a forced CXFS shutdown that forces a loss of quorum, the remaining part of the cluster (which now must also do an administrative stop) will not reset the departing node.

To perform an administrative stop, see

• “Revoke Membership of the Local Node with the GUI” in Chapter 10

• “Disable a Node with cxfs_admin” in Chapter 11

Node Status and Forced CXFS Shutdown

After a forced CXFS shutdown, the node is still considered part of the configured cluster and is taken into account when propagating the cluster database and when computing the cluster database (fs2d) membership quorum. (This could cause a loss of quorum for the rest of the cluster, causing the other nodes to do a forced CXFS shutdown). The state is INACTIVE in the GUI or inactive in cxfs_admin.

It is important that this node stays accessible and keeps running the cluster infrastructure daemons to ensure database consistency. In particular, if more than half the nodes in the pool are down or not running the infrastructure daemons, cluster database updates will stop being propagated and will result in inconsistencies. To be safe, you should remove those nodes that will remain unavailable from the cluster and pool.

Rejoining the Cluster after a Forced CXFS Shutdown

After a forced CXFS shutdown, the local node will not resume CXFS kernel membership until the node is rebooted or until you explicitly allow CXFS kernel membership for the local node. See:

• “Allow Membership of the Local Node with the GUI” in Chapter 10

• “Disable a Node with cxfs_admin” in Chapter 11

• “Enable a Node with cxfs_admin” in Chapter 11

If you perform a forced CXFS shutdown on a server-capable administration node, you must restart CXFS on that node before it can return to the cluster. If you do this while the cluster database still shows that the node is in the cluster and is activated, the node will restart the CXFS kernel membership daemon. Therefore, you may want to do this after resetting the database or after stopping CXFS services.

Reset Capability and a Forced CXFS Shutdown

---

Caution: If you perform an administrative CXFS stop on a server-capable administration node with system reset capability and the stop will not cause loss of cluster quorum, the node will be reset (rebooted) by the appropriate node.

---

For more information about resets, see “System Reset” in Chapter 2.

If the following chkconfig arguments are turned off, the clconfd and cxfs_client daemons on server-capable administration nodes and client-only nodes, respectively, will not be started at the next reboot and the kernel will not be configured to join the cluster:

It is useful to turn these arguments off before rebooting if you want to temporarily remove the nodes from the cluster for system or hardware upgrades or for other maintenance work.

For example, do the following on a server-capable administration node:

server-admin# /sbin/chkconfig grio2 off (If running GRIOv2)
server-admin# /sbin/chkconfig cxfs off
server-admin# /sbin/chkconfig cxfs_cluster off
server-admin# reboot

For more information, see “chkconfig Arguments”.

CXFS log files should be rotated at least weekly so that your disk will not become full.

A package that provides CXFS daemons also supplies scripts to rotate the log files for those daemons via logrotate. SGI installs the following scripts on server-capable administration nodes:

/etc/logrotate.d/cluster_admin
/etc/logrotate.d/cluster_control
/etc/logrotate.d/cxfs_cluster
/etc/logrotate.d/grio2

To customize log rotation, edit these scripts..

For information about log levels, see “Configure Log Groups with the GUI” in Chapter 10.

Although filesystem information is traditionally stored in /etc/fstab, the CXFS filesystems information is relevant to the entire cluster and is therefore stored in the replicated cluster database instead.

As the administrator, you will supply the CXFS filesystem configuration by using the CXFS GUI or cxfs_admin. The information is then automatically propagated consistently throughout the entire cluster. The cluster configuration daemon mounts the filesystems on each node according to this information, as soon as it becomes available.

A CXFS filesystem will be automatically mounted on all the nodes in the cluster. You can add a new CXFS filesystem to the configuration when the cluster is active.

Whenever the cluster configuration daemon detects a change in the cluster configuration, it does the equivalent of a mount -a command on all of the filesystems that are configured.

---

Caution: You must not modify or remove a CXFS filesystem definition while the filesystem is mounted. You must unmount it first and then mount it again after making the modifications.

---

This section discusses the following:

Mounting Filesystems

You supply mounting information with the CXFS GUI or cxfs_admin.

---

Caution: Do not attempt to use the mount command to mount a CXFS filesystem. Doing so can result in data loss and/or corruption due to inconsistent use of the filesystem from different nodes.

---

When properly defined and mounted, the CXFS filesystems are automatically mounted on each node by the local cluster configuration daemon, clconfd, according to the information collected in the replicated database. After the filesystems configuration has been entered in the database, no user intervention is necessary.

Mount points cannot be nested when using CXFS. That is, you cannot have a filesystem within a filesystem, such as /usr and /usr/home.

You must perform the backup of a CXFS filesystem from the active metadata server for that filesystem. The xfsdump and xfsrestore commands make use of special system calls that will only function on the active metadata server. The filesystem can have active clients during a dump process.

In a clustered environment, a CXFS filesystem may be directly accessed simultaneously by many CXFS clients and the active metadata server. A filesystem may, over time, have a number of metadata servers. Therefore, in order for xfsdump to maintain a consistent inventory, it must access the inventory for past dumps, even if this information is located on another node. SGI recommends that the inventory be made accessible by potential metadata server nodes in the cluster using one of the following methods:

---

Note: It is the /var/lib/xfsdump directory that should be shared, rather than the /var/lib/xfsdump/inventory directory. If there are inventories stored on various nodes, you can use xfsinvutil to merge them into a single common inventory, prior to sharing the inventory among the nodes in the cluster.

---

If you use I/O fencing and then make changes to your hardware configuration, you must verify that switch ports are properly enabled so that they can discover the WWPN of the HBA for I/O fencing purposes.

You must check the status of the switch ports involved whenever any of the following occur:

To check the status, use the following command on a server-capable administration node:

server-admin# hafence -v

If any of the affected ports are found to be disabled, you must manually enable them before starting CXFS on the affected nodes:

After the port is enabled, the metadata server will be able to discover the new (or changed) WWPN of the HBA connected to that port and thus correctly update the switch configuration entries in the cluster database.

This section provides an example of modifying a cluster to provide private network failover by using the cxfs_admin command.

Do the following:

For more information, see Chapter 11, “cxfs_admin Command”.

This section discusses removing and restoring cluster members for maintenance:

These procedures are the safest way to perform these tasks but in some cases are not the most efficient. They should be followed if you have been having problems using standard operating procedures (performing a stop/start of CXFS services or a simple host shutdown or reboot).

Manually Starting/Stopping CXFS

---

Note: If you are going to perform maintenance on a potential metadata server, you should first shut down CXFS services on it. Disabled nodes are not used in CXFS kernel membership calculations, so this action may prevent a loss of quorum.

---

On server-capable administration nodes, the service cxfs_cluster script will be invoked automatically during normal system startup and shutdown procedures. (On client-only nodes, the path to the cxfs_client script varies by platform; see CXFS 5 Administration Guide for SGI InfiniteStorage.) This script starts and stops the processes required to run CXFS.

To start up CXFS processes manually on a server-capable administration node, enter the following commands:

server-admin# service grio2 start (if running GRIOv2) server-admin# service cxfs_cluster start server-admin# service cxfs start

To stop CXFS processes manually on a server-capable administration node, enter the following commands:

server-admin# service grio2 stop (stops GRIOv2 daemons) server-admin# service cxfs stop (stops the CXFS server-capable administration node control daemon) server-admin# service cxfs_cluster stop (stops the cluster administration daemons)

---

Note: There is also a restart option that performs a stop and then a start.

---

admin1# /sbin/chkconfig grio2 off (if running GRIOv2)
admin1# /sbin/chkconfig cxfs off 
admin1# /sbin/chkconfig cxfs_cluster off 

The cxfs-enumerate-wwns script enumerates the worldwide names (WWNs) on the host that are known to CXFS. You can use the cxfs-enumerate-wwns script on a server-capable administration node to map XVM volumes to storage targets:

server-admin# /var/cluster/clconfd-scripts/cxfs-enumerate-wwns | grep -v "#"| sort -u

This section discusses the following:

XVM Failover Concepts

The example in Figure 12-6 shows two RAID controllers and the LUNs they own. All LUNs are visible from each controller, therefore, each LUN can be accessed by each path. However, the controller for RAID A is preferred for LUN 0 and LUN 2, and the controller for RAID B is preferred for LUN 1 and LUN 3.

Figure 12-6. Disk Paths

---

Note: The Mac OS X platform provides dynamic load balancing between all paths to the same RAID controller. In this case, the system will only show one path per controller to each LUN with local HBA ports or individual paths not visible.

---

MDS# xvm show -v phys | grep affinity > templatefile

/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port1/lun3/disc  affinity=1
/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port2/lun3/disc  affinity=1
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port1/lun3/disc  affinity=1
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port2/lun3/disc  affinity=1   preferred
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port1/lun3/disc  affinity=3
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port1/lun3/disc  affinity=3
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port2/lun3/disc  affinity=3
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port2/lun3/disc  affinity=3

/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port1/lun4/disc, affinity=1
/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port2/lun4/disc, affinity=1
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port1/lun4/disc, affinity=1
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port2/lun4/disc, affinity=1   
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port1/lun4/disc, affinity=3 
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port2/lun4/disc, affinity=3 preferred
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port1/lun4/disc, affinity=3
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port2/lun4/disc, affinity=3

/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port1/lun4/disc  affinity=1
/dev/xscsi/pci0004:00:01.1/node200900a0b813b982/port2/lun4/disc  affinity=2 
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port1/lun4/disc  affinity=1
/dev/xscsi/pci0004:00:01.0/node200900a0b813b982/port2/lun4/disc  affinity=2
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port1/lun4/disc  affinity=4
/dev/xscsi/pci0004:00:01.1/node200800a0b813b982/port2/lun4/disc  affinity=3 preferred
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port1/lun4/disc  affinity=4
/dev/xscsi/pci0004:00:01.0/node200800a0b813b982/port2/lun4/disc  affinity=3

/dev/dsk/200800a0b818b4de/lun4vol/c2p2 affinity=1 preferred
/dev/dsk/200800a0b818b4de/lun4vol/c2p1 affinity=1
/dev/dsk/200900a0b818b4de/lun4vol/c2p2 affinity=3
/dev/dsk/200900a0b818b4de/lun4vol/c2p1 affinity=3

XVM Commands Related to Failover V2

The following are useful XVM commands related to failover V2:

xvm help -verbose foconfig xvm help -verbose foswitch xvm help -verbose show xvm foconfig -init xvm foswitch -cluster -preferred physvol/name (switch phys/name in all nodes in cluster to preferred path) xvm foswitch -preferred physvol xvm foswitch -affinity 1 physvol xvm foswitch -dev newdev xvm foswitch -cluster -affinity 1 phys xvm foswitch -cluster -setaffinity X phys/name (switch phys/name in cluster to affinity "X") xvm show -verbose physvol xvm show -verbose physvol | fgrep affinity > templatefile

For details, see the XVM Volume Manager Administrator's Guide.

---

Note: The xvm command is provided on all CXFS platforms. However, client-only nodes support only read-only commands.

---

To generate streaming workload for SD/HD/2K/4K formats of video streams, you can use the frametest(1) command. Each frame is stored in a separate file. You can also use this tool to simulate the reading and writing video streams by streaming applications. The tool also generates the performance statistics for the reading and writing operation, so it can be very useful for performance analysis for streaming applications.

For example, to do a multithreaded (4 threads) write test of 20,000 HD frames, as fast as possible (the dir directory should contain 20,000 HD frames created by a previous write test):

# frametest -t4 -w hd -n20000 -x frametest_w_t4_hd_20000_flatout.csv dir

To use 24 frames per second using a buffer of 24 frames:

# frametest -t4 -n20000 -f24 -q24 -g frametest_r_t4_hd_20000_24fps_24buf.csv dir

For details about frametest and its command-line options, see the frametest(1)

---

Note: Using the frametest command on AIX requires that the posix_aio0 device is available. For more information, see the Solaris chapter in CXFS 5 Client-Only Guide for SGI InfiniteStorage.

---

The framesort utility provides easy file-layout analysis and advanced file-sequence reorganization:

File-layout analysis shows the following:

File-sequence reorganization makes files with consecutive filenames be placed consecutively in storage. It can also align files to their stripe-unit boundary. After rearrangement, files can can gain higher retrieval bandwidth, which is essential for frame playback.

---

Note: On IRIX systems, files larger than the specified maxdmasz size are not processed. (maxdmasz is the system tunable parameter that sets the maximum direct-memory access size). For more information about maxdmasz, see IRIX Admin: System Configuration and Operation.

---

For example, the following command line will do analysis and rearrangement recursively starting from directory movie1. It also displays the progress status and verbose information. If the percentage of poorly organized files is equal to or greater than 15%, the rearrangement is triggered:

# framesort -rdgvva 15 movie1

For details about command-line arguments, see the framesort(1) man page.

This section discusses the following:

Ideal Frame Layout

An ideal frame layout is one in which frames for each stream are written sequentially on disk to maximize bandwidth and minimize latency:

• Minimize seek times while reading and writing

• Maximize RAID prefetch into cache for reads

• Maximize RAID coalescing writes into larger writes to each disk

Figure 12-7 shows an ideal frame layout

Figure 12-7. Ideal Frame Layout

With multithreaded applications (such as frametest), there will be multiple requests in flight simultaneously. As each frame is requested, data from upcoming frames will be prefetched into cache. Figure 12-8 shows an example of a 4-thread frametest read (2-MB stripe unit / 1-GB cache size/ prefetch = x1 / 16 slices).

Figure 12-8. Ideal Frame Layout with RAID Prefetch

Multiple Streams of Real-Time Applications

When there are multiple streams of real-time applications, frames from each stream are interleaved into the same region. Frames are not written sequentially but will jump forwards and backwards in the filesystem. The RAID is unable to support many real-time streams and is unable to maintain frame rates due to additional back-end I/O. Filesystems allocate files based on algorithms to utilize free space, not to maximize RAID performance when reading streams back. Figure 12-9 shows an example.

Figure 12-9. Multiple Streams of Real Applications

Figure 12-10 shows an example of poor cache utilization.

Figure 12-10. Poor Cache Utilization

The filestreams Mount Option

Approved media customers can use the XFS filestreams mount option with CXFS to maximize the ability of storage to support multiple real-time streams of video data. It is appropriate for workloads that generate many files that are created and accessed in a sequential order in one directory.

---

Caution: SGI must validate that your RAID model and RAID configuration can support the use of the filestreams mount option to achieve real-time data transfer and that your application is appropriate for its use. Use of this feature is complex and is reserved for designs that have been approved by SGI.

---

The filestreams mount option changes the behavior of the XFS allocator in order to optimize disk layout. It selects an XFS disk block allocation strategy that does the following:

• Identifies streams writing into the same directory and locks down a region of the filesystem for that stream, which prevents multiple streams from using the same allocation groups

• Allocates the file data sequentially on disk in the order that the files are created, space permitting

• Uses different regions of the filesystem for files in different directories

Using the filestreams mount option can improve both bandwidth and latency when accessing the files because the RAID will be able to access the data in each directory sequentially. Therefore, multiple writers may be able to write into the same filesystem without interleaving file data on disk. Filesystem can be filled up to approximately 94% before performance degrades. Deletion of projects does not fragment filesystem, therefore there is no need to rebuild filesystem after each project.

You can safely enable the filestreams mount option on an existing filesystem and later disable it without affecting compatibility. (The mount option affects where data is located in the filesystem; it does not change the format of the filesystem.) However, you may not get the full benefit of filestreams due to preexisting filesystem fragmentation.

Figure 12-11 shows an example of excellent cache utilization that allows for more streams.

Figure 12-11. Excellent Cache Utilization

For more information, contact SGI Support.

CXFS has limited support for case-insensitive filesystems:

---

Note: Be aware that some applications rely on the case of filenames and will be confused when used with a case-insensitive filesystem.

---

Whether a CXFS filesystem is case-insensitive is determined when the filesystem is created. To create a case-insensitive filesystem, provide the following option to mkfs.xfs:

-n version=ci

For example:

# mkfs.xfs -n version=ci /dev/cxvm/tp91
meta-data=/dev/cxvm/tp91         isize=256    agcount=16, agsize=2746480 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=43943680, imaxpct=25
         =                       sunit=16     swidth=32 blks
naming   =version 2              bsize=4096   ascii-ci=1
log      =internal log           bsize=4096   blocks=21472, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=0
realtime =none                   extsz=4096   blocks=0, rtextents=0