When you install the CXFS software, you must modify certain system files. The network configuration is critical. Each node in the cluster must be able to communicate with every other node in the cluster by both logical name and IP address without going through any other network routing; proper name resolution is key. SGI recommends static routing.
This section provides an overview of the steps that you should perform on your nodes prior to installing the CXFS software. It contains the following sections:
 | Caution: It is critical that you understand these rules before attempting to configure a CXFS cluster. |
Use the following hostname resolution rules and recommendations when defining a node:
The first node you define in the pool must be an administration node.
Hostnames cannot begin with an underscore (_) or include any white-space characters.
The private network IP addresses on a running node in the cluster cannot be changed while CXFS services are active.
You must be able to communicate directly between every node in the cluster (including client-only nodes) using IP addresses and logical names, without routing.
You must dedicate a private network for control messages, CXFS metadata, CXFS kernel heartbeat messages, and cluster database heartbeat messages. No other load is supported on this network.
The private must be connected to all nodes, and all nodes must be configured to use the same subnet for that network.
Because CXFS kernel heartbeat and cluster database heartbeat are done using IP multicast, the private network must be multicast-capable. This means that all of the interfaces must have multicast enabled (which is the default) and all of the external networking hardware (such as switches) must support IP multicast.
If you change hostname resolution settings in the /etc/nsswitch.conf file after you have defined the first administration node (which creates the cluster database), you must re-create the cluster database.
The following procedure provides an overview of the steps required to add a private network.
 | Note: A private network is required for use with CXFS. |
You may skip some steps, depending upon the starting conditions at your site.
Edit the /etc/hosts file so that it contains entries for every node in the cluster and their private interfaces as well.
The /etc/hosts file has the following format, where primary_hostname can be the simple hostname or the fully qualified domain name:
IP_address primary_hostname aliases
You should be consistent when using fully qualified domain names in the /etc/hosts file. If you use fully qualified domain names on a particular node, then all of the nodes in the cluster should use the fully qualified name of that node when defining the IP/hostname information for that node in their /etc/hosts file.
The decision to use fully qualified domain names is usually a matter of how the clients are going to resolve names for their client/server programs (such as NFS), how their default resolution is done, and so on.
Even if you are using the domain name service (DNS) or the network information service (NIS), you must add every IP address and hostname for the nodes to /etc/hosts on all nodes. For example:
190.0.2.1 server1-example.com server1 190.0.2.3 stocks 190.0.3.1 priv-server1 190.0.2.2 server2-example.com server2 190.0.2.4 bonds 190.0.3.2 priv-server2
You should then add all of these IP addresses to /etc/hosts on the other nodes in the cluster.
For more information, see the hosts(5) and resolve.conf(5) man pages.
Note: Exclusive use of NIS or DNS for IP address lookup for the nodes will reduce availability in situations where the NIS or DNS service becomes unreliable.
Edit the /etc/nsswitch.conf file so that local files are accessed before either NIS or DNS. That is, the hosts line in /etc/nsswitch.conf must list files first.
For example:
hosts: files nis dns
(The order of nis and dns is not significant to CXFS, but files must be first.)
Configure your private interface according to the instructions in the Network Configuration section of your Linux distribution manual. To verify that the private interface is operational, use the ifconfig -a command.
For example:
server-admin# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:50:81:A4:75:6A inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:13782788 errors:0 dropped:0 overruns:0 frame:0 TX packets:60846 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:826016878 (787.7 Mb) TX bytes:5745933 (5.4 Mb) Interrupt:19 Base address:0xb880 Memory:fe0fe000-fe0fe038 eth1 Link encap:Ethernet HWaddr 00:81:8A:10:5C:34 inet addr:10.0.0.10 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:19 Base address:0xef00 Memory:febfd000-febfd038 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:162 errors:0 dropped:0 overruns:0 frame:0 TX packets:162 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11692 (11.4 Kb) TX bytes:11692 (11.4 Kb)
This example shows that two Ethernet interfaces, eth0 and eth1, are present and running (as indicated by UP in the third line of each interface description).
If the second network does not appear, it may be that a network interface card must be installed in order to provide a second network, or it may be that the network is not yet initialized.
(Optional) Make the modifications required to use CXFS connectivity diagnostics. See “Modifications for CXFS Connectivity Diagnostics” in Chapter 7.
For each private network on each Linux node in the pool, verify access with the ping command:
Enable multicast ping using one of the following methods (the permanent method will not take affect until after a reboot):
Immediate but temporary method:
linux# echo "0" > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
For more information, see http://kerneltrap.org/node/16225
Immediate but temporary method:
linux# sysctl -w net.ipv4.icmp_echo_ignore_broadcasts=0"
Permanent method upon reboot (survives across reboots):
Remove the following line (if it exists) from the /etc/sysctl.conf file:
net.ipv4.icmp_echo_ignore_broadcasts = 1
Add the following line to the /etc/sysctl.conf file:
net.ipv4.icmp_echo_ignore_broadcasts = 0
Execute a ping using the private network. Enter the following, where nodeIPaddress is the IP address of the node:
ping nodeIPaddress
For example:
linux# ping 10.0.0.1 PING 10.0.0.1 (10.0.0.1) from 128.162.240.141 : 56(84) bytes of data. 64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.310 ms 64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.122 ms 64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=0.127 ms
Execute a ping using the public network.
If ping fails, repeat the following procedure on each node:
Verify that the network interface was configured up using ifconfig. For example:
linux# ifconfig eth1 eth1 Link encap:Ethernet HWaddr 00:81:8A:10:5C:34 inet addr:10.0.0.10 Bcast:10.0.0.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:100 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Interrupt:19 Base address:0xef00 Memory:febfd000-febfd038
In the third output line above, UP indicates that the interface was configured up.
Verify that the cables are correctly seated.
Repeat this procedure on each node.
In order to test node connectivity by using the GUI, the root user on the node running the CXFS diagnostics must be able to access a remote shell using the rsh command (as root) on all other nodes in the cluster. (This test is not required when using cxfs_admin because it verifies the connectivity of each node as it is added to the cluster.)
There are several ways of accomplishing this, depending on the existing settings in the pluggable authentication modules (PAMs) and other security configuration files.
The following method works with default settings. Do the following on all nodes in the cluster:
After you have completed running the connectivity tests, you may wish to disable rsh on all cluster nodes.
For more information, see the Linux operating system documentation about PAM and the hosts.equiv man page.