 | Caution: CXFS is a complex product. To ensure that it is installed
and configured in an optimal manner, you must purchase
installation services from SGI. You should read through the following
chapters before attempting to install and configure a CXFS cluster: Also see the CXFS 7 Client-Only Guide for SGI InfiniteStorage. |
This chapter provides recommendations and a summary of the basic steps required to initially configure a cluster. It contains the following:
You should also refer to the information in “Configuration Best Practices” in Chapter 2 and you may wish to use the worksheet provided in Appendix J, “Initial Configuration Checklist”.
This chapter points to detailed descriptions in the task reference chapters and in the XVM Volume Manager Administrator Guide.
For information about licenses, see Chapter 5, “CXFS Licensing”.
Complete the following steps to ensure that you are ready to configure the initial cluster:
During the course of configuration, you will see various information-only messages in the log files. See “Normal Messages” in Chapter 15.
Verify that you have the appropriate CXFS licenses by using the cxfslicense -d command on server-capable administration nodes after installing the CXFS software. See “Displaying the Keys with cxfslicense After Installing CXFS” in Chapter 5.
Ensure that the appropriate chkconfig arguments are on. For more information, see “chkconfig Arguments” in Chapter 12.
Use chkconfig --list to verify that cxfs_cluster and cxfs are set to on for the site's normal run levels. For example, if the normal run levels are 3 and 5:
server-admin# /sbin/chkconfig --list | grep cxfs cxfs_cluster 0:off 1:off 2:off 3:on 4:off 5:on 6:off cxfs 0:off 1:off 2:off 3:on 4:off 5:on 6:off |
 | Note: Your site's normal run levels may differ. |
If the normal run levels are set to off, set them to on and reboot. For example:
server-admin# /sbin/chkconfig cxfs_cluster on server-admin# /sbin/chkconfig cxfs on server-admin# /sbin/chkconfig grio2 on (if running GRIOv2) server-admin# reboot |
When you first install the software, the following daemons should be running on all server-capable administration nodes:
| cad |
| cmond |
| crsd |
| fs2d |
| clconfd |
To determine which daemons are running on a server-capable administration node, enter the following:
server-admin# service cxfs_cluster status fs2d is running. cmond is running. cad is running. crsd is running. server-admin# service cxfs status clconfd is running. |
If you have previously enabled GRIOv2, the ggd2 daemon should also be running on a server-capable administration node.
If you do not see these processes on a server-capable administration node, see “Verify that the chkconfig Arguments are On”.
For more information, see:
You should know the fully qualified hostname of the machine from which you will do CXFS administration, which should be the first node you define in the cluster database. If you use cxfs_admin (see “Initial Setup”), you should use the hostname when defining the first node. (This information is automatically supplied for you in the CXFS GUI.)
You should also know the IP addresses and hostnames of the other machines that will form the cluster and the name by which you want to refer to the cluster.
You can create the cluster and its components using either of the following tools, which provide similar functionality:
| Caution: You should only use one configuration tool at a time to make changes. |
The following procedures provide an overview of the basic steps to set up a cluster. You will first define a server-capable administration node from which you perform administrative tasks, and then the other components of the cluster.
| Note: For complete details about using the GUI, see “CXFS Tools” in Chapter 1 and Chapter 10, “CXFS GUI”. |
To initially configure the cluster with GUI, do the following:
The server-capable administration node to which you connect the GUI affects your view of the cluster. You should wait for a change to appear in the view area before making another change; the change is not guaranteed to be propagated across the cluster until it appears in the view area. You should only make changes from one instance of the GUI at any given time; changes made by a second GUI instance may overwrite changes made by the first instance.
Start the CXFS Manager by entering the following:
server-admin# /usr/bin/cxfsmgr |
You can also start the GUI from your web browser. For more information, see “Starting the GUI via the Command Line” in Chapter 10.
Supply the name of the server-capable administration node you wish to connect to and the root password.
Figure 9-1 shows an example of the GUI.
Within the CXFS tasks, you can click any blue text to get more information about that concept or input field. In every task, the cluster configuration will not update until you click on OK.
| Note: To specify reset method that uses Intelligent Platform Management Interface (IPMI) and baseboard management controller (BMC), you must use the cxfs_admin configuration tool. See “Create or Modify a Node with cxfs_admin” in Chapter 11. |
The Set Up a New Cluster task in the Guided Configuration menu leads you through the steps required to create a new cluster. It encompasses tasks that are detailed elsewhere.
Do the following:
Click Define a Node to define the server-capable administration node to which you are connected. See “Define a Node with the GUI” in Chapter 10.
Note: If you attempt to define a cluster or other object before the local node has been defined, you will get an error message that says: No nodes are registered on servername. You cannot define a cluster until you define the node to which the GUI is connected. To do so, click "Continue" to launch the "Set Up a New Cluster" task.
(Optional) After the first node icon appears in the view area on the left, click step 2, Define a Node, to define the other nodes in the cluster. To use private network failover, you must use the cxfs_admin command's create failover_net command to specify the network and mask; see “Network Failover Modification Tasks with cxfs_admin” in Chapter 11. See “Define a Node with the GUI” in Chapter 10.
Note: Do not define another node until this node appears in the view area. If you add nodes too quickly (before the database can include the node), errors will occur.
Repeat this step for each node. For large clusters, define only the server-capable administration nodes first; see “Configuring a Large Cluster”.
Click Define a Cluster to create the cluster definition. See “Define a Cluster with the GUI” in Chapter 10. Verify that the cluster appears in the view area. Choose View: Nodes and Cluster.
After the cluster icon appears in the view area:
Click Add/Remove Nodes in Cluster to add the server-capable administration nodes to the new cluster. Select the server-capable administration node names and click Add to add them to the list and click OK to complete the task. See “Add or Remove Nodes in the Cluster with the GUI” in Chapter 10.
Note: SGI recommends that you form an initial cluster that consists of all of the server-capable administration nodes, to ensure that these nodes will have lower cell ID numbers than any client-only nodes. See “Create an Initial Cluster of All Server-Capable Administration Nodes” in Chapter 2. Click Add/Remove Nodes in Cluster to add the client-only nodes to the new cluster. Select the client-only node names and click Add to add them to the list and click OK to complete the task. See “Add or Remove Nodes in the Cluster with the GUI” in Chapter 10.
Note: For large clusters, SGI recommends that you build up the remainder of the cluster in small groups of client-only nodes. This makes it easier to locate and fix problems, should any occur. See “Configuring a Large Cluster”.
(Optional) Click on Test Connectivity to verify that the nodes are physically connected. See “Test Node Connectivity with the GUI” in Chapter 10. (This test requires the proper configuration; see “Modifications for CXFS Connectivity Diagnostics” in Chapter 7.)
If you are using I/O fencing, define the switch in the cluster; see the release notes for supported switches. I/O fencing is required for nodes without system controllers; see “Hardware and Software Requirements for Server-Capable Administration Nodes” in Chapter 1.
Click Start CXFS Services . See “Start CXFS Services with the GUI” in Chapter 10.
Click Close. Clicking on Close exits the task; it does not undo the task.
| Note: Within the CXFS tasks, you can click any blue text to get more information about that concept or input field. In every task, the cluster configuration will not update until you click OK. |
The Set Up a New CXFS Filesystem task leads you through the steps required to create a new filesystem and mount it on all nodes in your cluster. It encompasses tasks that are detailed elsewhere.
Do the following:
Click Start CXFS Services if the services have not been started already. (The current status is displayed beneath the task link.) See “Start CXFS Services with the GUI” in Chapter 10.
Click Label Disks.
Note: The disk must be initialized before being labeled. If your disk has not been initialized during factory set-up, use the Linux parted command to initialize the disk. For more information, see the XVM Volume Manager Administrator Guide.
Create slices, which define the physical storage, on the labeled disk. Click Slice Disks.
Create the type of volume you want (such as stripe or concat).
Note: Mirrors do not perform well in a CXFS cluster. See “Use RAID Mirroring Not XVM Mirroring” in Chapter 2. Click Make the Filesystem. If you do not want to use the default options, click Specify Sizes and go to the next page. For more information, see the mkfs(8) man page and the XVM Volume Manager Administrator Guide.
Click Define a CXFS Filesystem. This task lets you define a new filesystem, set the ordered list of potential metadata servers, and set the list of client nodes for the filesystem. See “Define CXFS Filesystems with the GUI” in Chapter 10.
Click Mount a CXFS Filesystem. This task lets you mount the filesystem on all nodes in the cluster. See “Mount CXFS Filesystems with the GUI” in Chapter 10.
Repeat these steps for each filesystem.
| Note: For the initial installation, SGI highly recommends that you use the GUI guided configuration tasks. See “Initial Setup with the CXFS GUI”. For complete details about using cxfs_admin, see “CXFS Tools” in Chapter 1 and Chapter 11, “cxfs_admin Command”. |
You can perform configuration with cxfs_admin using normal mode (in which you specify each command and attribute) or in prompting mode, in which cxfs_admin asks you for the information it requires.
To initially configure the cluster with cxfs_admin, do the following (line breaks shown here for readability). A simple example of prompting mode follows the steps.
Initialize the cluster database and start cxfs_admin:
server-admin# /usr/cluster/bin/cxfs_admin -s
Define the cluster name, where clustername is the logical name of the cluster:
cxfs_admin> create cluster name=clustername
For example:
cxfs_admin> create cluster name=mycluster
Create the first server-capable administration node (normally the node on which you are currently running cxfs_admin ). (You do not need to specify the node type because it must be server_admin.) If you use prompting mode, the name of the local node is used as a default for name.
Caution: It is critical that you enter the primary hostname for the first node defined in the pool. cxfs_admin> create node name=server_capable_hostname private_net=private_IPaddress
For example:
cxfs_admin> create node name=server1 private_net=10.11.20.114
Exit cxfs_admin and restart the CXFS cluster services:
server-admin# service grio2 stop (if running GRIOv2) server-admin# service cxfs stop server-admin# service cxfs_cluster stop server-admin# service cxfs_cluster start server-admin# service cxfs start server-admin# service grio2 start (if running GRIOv2)
Restart cxfs_admin:
server-admin# /usr/cluster/bin/cxfs_admin [-i clustername]
Note: If you have multiple clusters connected to the same public network, use the -i option to identify the cluster name.
(Optional) Create the failover networks:
cxfs_admin:cluster> create failover_net network=IPaddress1 mask=netmask cxfs_admin:cluster> create failover_net network=IPaddress2 mask=netmask
For example, for IPv4:
cxfs_admin:mycluster> create failover_net network=192.168.0.0 mask=255.255.255.0 cxfs_admin:mycluster > create failover_net network=10.0.0.0 mask=255.255.255.0
For example, for IPv6:
cxfs_admin:mycluster> create failover_net network=2001:db8::2:1 mask=64 cxfs_admin:mycluster > create failover_net network=fe80:: mask=64
Note: For IPv4, you can use either an IP address or the number of bits for the netmask. For IPv6, you must supply the netmask as the prefix number of bits. Create the switches:
cxfs_admin:cluster> create switch name=switchname [vendor=brocade|qlogic|voltaire|lsi] [user=username password=password]
For example:
cxfs_admin:mycluster> create switch name=myswitch vendor=qlogic
Create another server-capable administration node:
cxfs_admin:mycluster> create node name=nodename os=OStype private_net=IPaddress type=server_admin
For example:
cxfs_admin:mycluster> create node name=server2 os=Linux private_net=10.11.20.115 type=server_admin
Repeat this step for each server-capable administration node.
Note: SGI recommends that you form an initial cluster that consists of all of the server-capable administration nodes to ensure that these nodes will have lower cell ID numbers than any client-only nodes and thereby avoid potential problems. See “Create an Initial Cluster of All Server-Capable Administration Nodes” in Chapter 2. Create the client-only nodes:
cxfs_admin:mycluster> create node name=nodename [os=OStype] private_net=IPaddress [type=client_only]
For example:
cxfs_admin:mycluster> create node name=client1 os=Windows private_net=10.11.20.116
(Optional) Define one of the client-only nodes as the CXFS tiebreaker if using multiple server-capable administration nodes:
cxfs_admin:cluster> modify clustername tiebreaker=client_only_nodename
For example:
cxfs_admin:mycluster> modify mycluster tiebreaker=client1
Obtain a shell window for one of the server-capable administration nodes in the cluster and use the Linux parted command to create a volume header on the disk drive. For information, see the parted(8) man page and Linux Configuration and Operations Guide.
Create the XVM logical volumes. In the shell window, use the xvm command line interface. For information, see the XVM Volume Manager Administrator Guide.
Make the XFS filesystems. In the shell window, use the mkfs command. For information, see:
the .XFS Administrator Guide
XVM Volume Manager Administrator Guide
Create the CXFS filesystems:
cxfs_admin:cluster> create filesystem name=XVMvolume [mountpoint=path] [options=mount_options]
For example:
cxfs_admin:cluster> create filesystem name=cxfsvol1
(Optional) Create private network failover:
cxfs_admin:cluster> network=IPaddress mask=NetMask
For example, to create two private networks, one on the 192.168.0. x and the other on the 10.0.0.x subnets:
cxfs_admin:mycluster> create failover_net network=192.168.0.0 mask=255.255.255.0 cxfs_admin:mycluster > create failover_net network=10.0.0.0 mask=255.255.255.0
View the cluster status:
cxfs_admin:cluster> status
Following is a simple example using prompting mode:
cxfsopus14:~ # /usr/cluster/bin/cxfs_admin -s Event at [ Oct 26 11:38:00 ] Connecting to the local CXFS server... cxfs_admin:(no cluster defined)> create cluster Specify the attributes for create cluster: name? mycluster Event at [ Oct 26 11:38:04 ] cxfs_admin:mycluster> create node Specify the attributes for create node: name? cxfsopus14 type? server_admin private_net? 10.11.20.114 Event at [ Oct 26 11:38:08 ] Node "cxfsopus14" has been created, waiting for it to join the cluster... Please restart all cxfs and cluster services on the server "cxfsopus14" to make it join the cluster. Event at [ Oct 26 11:38:10 ] cxfs_admin:mycluster> create filesystem Specify the attributes for create filesystem: name? thump options? forced_unmount? false mountpoint? /mnt/thump mounted? true grio_managed? false Event at [ Oct 26 11:39:04 ] Filesystem "thump" has been created, waiting for it to be mounted on all assigned nodes... |
For more information, see Chapter 11, “cxfs_admin Command” and the help command within cxfs_admin.
When configuring a large cluster, you should ensure that a small cluster containing just the server-capable administration nodes is fully functional before adding any client-only nodes. By building up a large cluster with small groups of client-only nodes (step 3), you will minimize concurrent operational issues and use the database most efficiently.
Following is an overview of the procedure:
Create the initial cluster with just the server-capable administration nodes and test it:
Define the cluster name.
Define the initial server-capable administration node.
(Optional) Create the failover networks.
Create the switches.
Define all of the other the server-capable administration nodes.
If using the GUI, add all of the server-capable administration nodes to the cluster. (In cxfs_admin, nodes that are created are automatically added to the cluster.)
Verify that the nodes are all part of the cluster membership.
Add the client-only nodes to the database:
Define all client-only nodes.
Add all client-only nodes to the cluster. (In cxfs_admin, nodes that are created are automatically added to the cluster.).
Gradually build up the functional cluster with subsets of client-only nodes:
Repeat step 3 as needed to complete the cluster membership.
Create the CXFS filesystems.
(Optional) Manually mount the CXFS filesystems.
Verify that the nodes are all part of the cluster membership and that all of the filesystems are mounted and fully functional.
Following is an example cxfs_admin script to configure a cluster. The first node line creates the first server-capable administration node; you can copy and repeat the second node line for each remaining server-capable or client-only node in the cluster:
| Note: When using a script, SGI strongly recommends using mounted=false and then mounting the filesystems manually. |
create cluster name=clustername create node name=nodename private_net=IPaddress [type=server_admin] create failover_net network=IPaddress1 mask=netmask create switch name=switchname [vendor=brocade|qlogic|voltaire|lsi] [user=username password=password] [copy and repeat] create node name=nodename type=server_admin private_net=IPaddress [copy and repeat] create node name=nodename [os=OS] private_net=IPaddress [type=client_only] [copy and repeat] create filesystem name=filesystemname forced_unmount=false mountpoint=/mnt/nodename mounted=false [copy and repeat] |
This section discusses the following:
For each private network on each node in the pool, enter the following, where IPaddress is the IP address of the destination node:
IPv4:
# ping -c 3 IPaddress
# ping -c3 128.162.240.71 PING bert.americas.sgi.com (128.162.240.71): 56 data bytes 64 bytes from 128.162.240.71: icmp_seq=0 ttl=62 time=2.181 ms 64 bytes from 128.162.240.71: icmp_seq=1 ttl=62 time=1.327 ms 64 bytes from 128.162.240.71: icmp_seq=2 ttl=62 time=1.189 ms ----bert.americas.sgi.com PING Statistics---- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max = 1.189/1.566/2.181 ms
IPv6:
# ping6 -I interface IPaddress
For example:
# ping6 -I eth1 -c 3 fe80::225:90ff:fe21:3087 PING fe80::225:90ff:fe21:3087(fe80::225:90ff:fe21:3087) from fe80::225:90ff:fe20:359 eth1: 56 data bytes 64 bytes from fe80::225:90ff:fe21:3087: icmp_seq=1 ttl=64 time=0.155 ms 64 bytes from fe80::225:90ff:fe21:3087: icmp_seq=2 ttl=64 time=0.179 ms 64 bytes from fe80::225:90ff:fe21:3087: icmp_seq=3 ttl=64 time=0.175 ms --- fe80::225:90ff:fe21:3087 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1999ms rtt min/avg/max/mdev = 0.155/0.169/0.179/0.018 ms
If ping fails, follow these steps:
Verify that the network interface was configured up by using ifconfig. For example:
# ifconfig ec3 ec3: flags=c63<UP,BROADCAST,NOTRAILERS,RUNNING,FILTMULTI,MULTICAST> inet 190.x.x.x netmask 0xffffff00 broadcast 190.x.x.x
The UP in the first line of output indicates that the interface was configured up.
Verify that the cables are correctly seated.
Repeat this procedure on each node.
To test the system reset connections, do the following:
Ensure that the nodes are powered on.
Start the cxfs_admin command as root in write mode (-A) on one of the server-capable administration nodes:
server-admin# /usr/cluster/bin/cxfs_admin -A [-i clustername]
Test the connections by entering of the following for each node:
cxfs_admin:clustername> control target-nodename operation=ping
If a command fails, make sure all the cables are seated properly and rerun the command.