The primary difference between XFS and CXFS filesystems is the way in which filesystems are mounted and managed:

XFS features that are also present in CXFS include the following:

CXFS preserves these underlying XFS features while distributing the I/O directly between the LUNs and the nodes. The efficient XFS I/O path uses asynchronous buffering techniques to avoid unnecessary physical I/O by delaying writes as long as possible. This allows the filesystem to allocate the data space efficiently and often contiguously. The data tends to be allocated in large contiguous chunks, which yields sustained high bandwidths.

The XFS directory structure is based on B-trees, which allow XFS to maintain good response times even as the number of files in a directory grows to tens or hundreds of thousands of files.

You should use CXFS when you have multiple nodes running applications that require high-bandwidth access to common filesystems. CXFS performs best under the following conditions:

For most filesystem loads, the scenarios above represent the bulk of the file accesses. Thus, CXFS delivers fast local file performance. CXFS is also useful when the amount of data I/O is larger than the amount of metadata I/O. CXFS is faster than NFS because the data does not go through the network.

CXFS may not give optimal performance under the following circumstances, and you should give extra consideration to using CXFS in these cases:

Accessing remote files over local area networks (LANs) can be significantly slower than accessing local files. The network hardware and software introduces delays that tend to significantly lower the transaction rates and the bandwidth. These delays are difficult to avoid in the client-server architecture of LAN-based network filesystems. The delays stem from the limits of the LAN bandwidth, latency, and shared path through the data server.

LAN bandwidths force an upper limit for the speed of most existing shared filesystems. This can be one to several orders of magnitude slower than the bandwidth possible across multiple disk channels to local or shared disks. The layers of network protocols and server software also tend to limit the bandwidth rates.

A shared fileserver can be a bottleneck for performance when multiple clients wait their turns for data, which must pass through the centralized fileserver. For example, NFS and Samba servers read data from disks attached to the server, copy the data into UDP/IP or TCP/IP packets, and then send it over a LAN to a client host. When many clients access the server simultaneously, the server's responsiveness degrades.

A CXFS filesystem is a clustered XFS filesystem that allows for logical file sharing similar to network filesystems, but with significant performance and functionality advantages. CXFS runs on top of a SAN, where each node in the cluster has direct high-speed data channels to a shared set of disks.

This section discusses the following:

Metadata is information that describes a file, such as the file's name, size, location, and permissions. Metadata tends to be small, usually about 512 bytes per file in XFS. This differs from the data, which is the contents of the file. The data may be many megabytes or gigabytes in size.

A node is an operating system (OS) image, usually an individual computer. (This use of the term node does not have the same meaning as a node in an SGI Origin 3000 or SGI 2000 system.)

A redundant array of independent disks.

A logical unit number (LUN) is a representation of disk space. In a RAID, the disks are not individually visible because they are behind the RAID controller. The RAID controller will divide up the total disk space into multiple LUNs. The operating system sees a LUN as a hard disk. A LUN is what XVM uses as its physical volume (physvol). For more information, see the XVM Volume Manager Administrator's Guide.

A cluster is the set of nodes configured to work together as a single computing resource. A cluster is identified by a simple name and a cluster identification (ID) number. A cluster running multiple operating systems is known as a multiOS cluster . A given node can be a member of only one cluster.

LUNs are assigned to a cluster by recording the name of the cluster on the LUN. Thus, if any LUN is accessible (via a Fibre Channel connection) from nodes in different clusters, then those clusters must have unique names. When members of a cluster send messages to each other, they identify their cluster via the cluster ID. Cluster names and IDs must be unique.

Because of the above restrictions on cluster names and IDs, and because cluster names and IDs cannot be changed after the cluster is created (without deleting the cluster and recreating it), SGI advises that you choose unique names and IDs for each of the clusters within your organization.

The pool is the set of nodes from which a particular cluster may be formed. (All nodes created when using the cxfs_admin tool are automatically part of the cluster, so the concept of the pool is obsolete when using cxfs_admin.)

Only one cluster may be configured from a given pool, and it need not contain all of the available nodes. (Other pools may exist, but each is disjoint from the other. They share no node or cluster definitions.)

A pool is first formed when you connect to a given server-capable administration node (see “Node Types, Node Functions, and the Cluster Database”) and define that node in the cluster database using the GUI. You can then add other nodes to the pool by defining them while still connected to the first node. (If you were to connect to a different server-capable administration node and then define it, you would be creating a second pool).

Figure 1-1 shows the concepts of pool and cluster. Node9 and Node10 have been defined as nodes in the CXFS pool, but they have not been added to the cluster definition.

The cluster database (CDB) contains configuration and logging information. A node is defined in the cluster database as one of the following types:

For each CXFS filesystem, one server-capable administration node is responsible for updating that filesystem's metadata. This node is referred to as the metadata server.

Multiple server-capable administration nodes can be defined as potential metadata servers for a given CXFS filesystem, but only one node per filesystem is chosen to function as the active metadata server, based on various factors. There can be multiple active metadata servers in the cluster, one per CXFS filesystem.

All other nodes that mount a CXFS filesystem function as CXFS clients. A server-capable administration node can function at any point in time as either an active metadata server or a CXFS client, depending upon how it is configured and whether it is chosen to function as the active metadata server.

The metadata server must perform cluster-coordination functions such as the following:

All CXFS requests for metadata are routed over a TCP/IP network and through the metadata server, and all changes to metadata are sent to the metadata server. The metadata server uses the advanced XFS journal features to log the metadata changes. Because the size of the metadata is typically small, the bandwidth of a fast Ethernet local area network (LAN) is generally sufficient for the metadata traffic.

The operations to the CXFS metadata server are typically infrequent compared with the data operations directly to the LUNs. For example, opening a file causes a request for the file information from the metadata server. After the file is open, a process can usually read and write the file many times without additional metadata requests. When the file size or other metadata attributes for the file change, this triggers a metadata operation.

The following rules apply:

Figure 1-2 shows nodes in a pool that are installed with the cluster_admin software product and others that are installed with the cxfs_client software product. Only those nodes with the cluster_admin software product have the fs2d daemon and therefore a copy of the cluster database. (For more information, see Table 1-2 and Table 1-3.)

A standby node is a server-capable administration node that is configured as a potential metadata server for a given filesystem.

Ideally, all server-capable administration nodes will run the same version of the operating system. However, SGI supports a policy for CXFS that permits a rolling upgrade; see “CXFS Release Versions and Rolling Upgrades” in Chapter 12.

The following figures show a few configuration possibilities. The potential metadata servers are required to be server-capable administration nodes; the other nodes should be client-only nodes.

Figure 1-3 shows a very simple configuration with a single metadata server and no potential (standby) metadata servers; recovery and relocation do not apply to this cluster. The database exists only on the server-capable administration node. The client-only nodes could be running any supported operating system.

Figure 1-4 shows a configuration with two server-capable administration nodes, both of which are potential metadata servers for filesystems /a and /b:

Neither Node1 nor Node2 runs applications because they are both potential metadata servers. For simplicity, the figure shows one client-only node, but there could be many. One client-only node should be the tiebreaker so that the configuration remains stable if one of the server-capable administration nodes is disabled.

Figure 1-5 shows three server-capable administration nodes as active metadata servers. Node1 is the active metadata server for both filesystems /a and /b. If Node1 failed, Node2 would take over as the active metadata server according to the list of potential metadata servers for filesystems /a and /b.

The nodes in a cluster must act together to provide a service. To act in a coordinated fashion, each node must know about all the other nodes currently active and providing the service. The set of nodes that are currently working together to provide a service is called a membership:

CXFS kernel heartbeat messages are exchanged via a private network so that each node can verify each membership:

A heartbeat timeout occurs when a node does not receive a heartbeat packet from another node within a predetermined period of time. Timeouts can happen in either direction, although they most frequently are seen when the server-capable administration node fails to receive a multicast heartbeat packet from a client-only node.

A private network is one that is dedicated to cluster communication and is accessible by administrators but not by users.

CXFS uses the private network for the following:

Even small variations in heartbeat timing can cause problems. If there are delays in receiving heartbeat messages, the cluster software may determine that a node is not responding and therefore revoke its CXFS kernel membership; this causes it to either be reset or disconnected, depending upon the configuration.

Rebooting network equipment can cause the nodes in a cluster to lose communication and may result in the loss of CXFS kernel membership and/or cluster database membership; the cluster will move into a degraded state or shut down if communication among nodes is lost. Using a private network limits the traffic on the network and therefore will help avoid unnecessary resets or disconnects. Also, a network with restricted access is safer than one with user access because the messaging protocol does not prevent snooping (illicit viewing) or spoofing (in which one machine on the network masquerades as another).

Therefore, because the performance and security characteristics of a public network could cause problems in the cluster and because CXFS kernel heartbeat and cluster database heartbeat are very timing-dependent, a private network is required. The private network should be used for metadata traffic only. (Although the primary network must be private, the backup network may be public.)

The private network must be connected to all nodes, and all nodes must be configured to use the same subnet for that network.

For more information, see:

A failed node must be isolated from the shared filesystems so that it cannot corrupt data. CXFS uses the following methods in various combinations to isolate failed nodes:

For more information, see “Protect Data Integrity on All Nodes” in Chapter 2,

The CXFS tiebreaker node is used in the process of computing the CXFS kernel membership for the cluster when exactly half the server-capable administration nodes in the cluster are up and can communicate with each other.

The tiebreaker is required for all clusters with more than one server-capable administration node and at least one client-only node. You should choose a reliable client-only node as the tiebreaker; there is no default. For a cluster that consists of only four or more server-capable administration nodes, you should choose one of them as the tiebreaker; this is the only situation in which you should choose a server-capable administration node as a tiebreaker.

The tiebreaker is required in addition to I/O fencing or system reset; see “Data Integrity Protection”.

Relocation is the process by which the metadata server moves from one node to another due to an administrative action; other services on the first node are not interrupted.

CXFS kernel membership is not affected by relocation. However, users may experience a degradation in filesystem performance while the metadata server is relocating.

An example of a relocation trigger is when the system administrator uses the GUI or cxfs_admin to relocate the metadata server.

To use relocation, you must enable relocation on the active metadata server. See “CXFS Relocation Capability” in Chapter 12.

See also “Use Relocation and Recovery Properly” in Chapter 2.

Recovery is the process by which the metadata server moves from one node to another due to an interruption in services on the first node. If the node acting as the metadata server for a filesystem dies, another node in the list of potential metadata servers will be chosen as the new metadata server. This assumes that at least two potential metadata servers are listed when you define a filesystem.

The metadata server that is chosen must be a filesystem client; other filesystem clients will experience a delay during the recovery process. Each filesystem will take time to recover, depending upon the number of active inodes; the total delay is the sum of time required to recover each filesystem. Depending on how active the filesystems are at the time of recovery, the total delay could take up to several minutes per filesystem.

If a CXFS client fails, the metadata server will clean up after the client. Other CXFS clients may experience a delay during this process. A delay depends on what tokens, if any, that the deceased client holds. If the client has no tokens, then there will be no delay; if the client is holding a token that must be revoked in order to allow another client to proceed, then the other client will be held up until recovery returns the failed nodes tokens (for example, in the case where the client has the write token and another client wants to read). The actual length of the delay depends upon the following:

The CXFS client that failed is not allowed to rejoin the CXFS kernel membership until all metadata servers have finished cleaning up after the client.

The following are examples of recovery triggers:

Figure 1-6 describes the difference between relocation and recovery for a metadata server. (Remember that there is one active metadata server per CXFS filesystem. There can be multiple active metadata servers within a cluster, one for each CXFS filesystem.)

See also “Use Relocation and Recovery Properly” in Chapter 2.

To start CXFS services enables a node, which changes a flag in the cluster database by performing an administrative task using the CXFS GUI or cxfs_admin.

To stop CXFS services disables a node, which changes a flag in the cluster database, by performing an administrative task using the GUI or cxfs_admin:

To shutdown CXFS services withdraws a node from the CXFS kernel membership, either due to the fact that the node has failed somehow. The node remains enabled in the cluster database.

Starting, stopping, or shutting down CXFS services does not affect the daemons involved. For information about the daemons that control CXFS services, see “CXFS Control Daemons”.

CXFS supports the use of hierarchical storage management (HSM) products through the data management application programming interface (DMAPI), also know as X/Open Data Storage Management Specification (XSDM). An example of an HSM product is the Data Migration Facility (DMF). DMF is the only HSM product supported with CXFS.

If DMF is managing a CXFS filesystem, DMF will ensure that the filesystem's CXFS metadata server is the DMF server and will use metadata server relocation if necessary to achieve that configuration. CXFS client-only nodes may be DMF parallel data mover nodes or DMF clients. For more information, see:

Linux-HA Heartbeat from the Linux-HA project provides high-availability services for CXFS clusters. The CXFS plug-in reports on whether the metadata server for a filesystem on a given node is running, not running, or has an error. Heartbeat then moves services accordingly, based upon constraints that the customer sets. For more information about the CXFS plug-in, see SGI InfiniteStorage High Availability Using Linux-HA Heartbeat. For more information about Heartbeat, see:

CXFS supports XVM labels on LUNs with GUID partition table (GPT) labels as well LUNs with SGI disk volume header (DVH) labels. A CXFS cluster can contain LUNs that have GPT labels and LUNs that have DVH labels. You can create these labels on server-capable administration nodes and Linux client-only nodes.

For information about creating GPT labels, see the XVM Volume Manager Administrator's Guide.

CXFS supports guaranteed-rate I/O (GRIO) version 2 clients on all platforms, with a GRIO server on a server-capable administration node. GRIO is disabled by default on the server-capable administration nodes and on Linux client-only nodes.

As the superuser, you can run the following commands from any node in the cluster:

Run the above tools with the -h (help) option for a full description of all available options.

The paths to the GRIO commands differ by platform. See Appendix C, “Path Summary” and the appendix in the CXFS 5 Client-Only Guide for SGI InfiniteStorage. For details about GRIO, see the Guaranteed-Rate I/O Version 2 for Linux Guide. For other platform-specific limitations and considerations, see the CXFS 5 Client-Only Guide for SGI InfiniteStorage.

This section describes the CXFS differences for the following:

CXFS uses the XVM volume manager. XVM can combine many LUNs into high transaction rate, high bandwidth, and highly reliable filesystems. CXFS uses XVM to provide the following:

For more information, see the XVM Volume Manager Administrator's Guide.

CXFS supports XVM failover V2 on all platforms. XVM failover V2 allows XVM to use multiple paths to LUNs in order to provide fault tolerance and static load balancing. Paths to LUNs are representations of links from the client HBA ports to the fabric and from the fabric to the RAID controller; they do not represent the path through the fabric itself.

SGI requires SGIAVT mode for XVM failover V2 because failover V2 is unable to issue the device-specific commands necessary to change LUN ownership when the RAID is in SGIRDAC mode. See “Changing SGIRDAC Mode to SGIAVT Mode for SGI RAID” in Appendix H.

In general, you want to evenly distribute the I/O to LUNs across all available host bus adapters and RAID controllers and attempt to avoid blocking in the SAN fabric. The ideal case, from a performance standpoint, is to use as many paths as connection endpoints between two nodes in the fabric as possible with as few blocking paths as possible in the intervening SAN fabric.

For more information, see “XVM Failover V2” in Chapter 12.

You can perform CXFS configuration tasks using the GUI or cxfs_admin , shown in Table 1-1. These tools update the cluster database, which persistently stores metadata and cluster configuration information. After the associated changes are applied to all online database copies in the pool, the view area in the GUI will be updated. You can use the GUI or the cxfs_admin command to view the state of the database. (The database is a collection of files, which you cannot access directly.)

The rest of this section discusses:

Table 1-3 lists the set of daemons that provide cluster infrastructure on a server-capable administration node.

You can start these daemons manually or automatically upon reboot by using the chkconfig command. For more information, see:

Table 1-3 lists the daemons that control CXFS nodes.

You can start these daemons manually or automatically upon reboot by using the chkconfig command. For more information, see:

Table 1-4 summarizes the other CXFS commands of most use on a server-capable administration node. For information about client-only nodes, see CXFS 5 Client-Only Guide for SGI InfiniteStorage.