This chapter discusses the following:
To use the cxfs_admin command, you must be logged in to a node that has permission to access the CXFS cluster database. See “Setting cxfs_admin Access Permissions”.
To make configuration changes using cxfs_admin, you must be logged in as root and execute cxfs_admin on a node with admin permission and in write mode. To perform some operational tasks, you must in addition execute cxfs_admin directly on a server-capable administration node. See “Making Changes Safely”
 | Note: For the steps to create a cluster for the first time, see “Initial Setup with the cxfs_admin Command” in Chapter 9. |
This section discusses the following:
To execute cxfs_admin in the default read-only mode, enter the following:
# /usr/cluster/bin/cxfs_admin [-i clustername] |
To execute cxfs_admin in write mode, enter the following:
# /usr/cluster/bin/cxfs_admin -A [-i clustername] |
| Note: If you have multiple clusters connected to the same public network, use the -i option to identify the cluster name. |
If no other user already holds the cxfs_admin lock, you will be given the lock by default when executing cxfs_admin in write mode. You can make executing cxfs_admin in write mode the default by setting the write_mode parameter in the $HOME/.cxfs_admin file; see “Setting cxfs_admin Defaults”.
See also:
Some cxfs_admin commands affect the cxfs_admin operating environment itself, some display status information, and others affect objects or classes. Within cxfs_admin, an object is a specific item that is configured in the CXFS cluster and a class contains a group of similar objects. For example, the filesystem names fs1 and fs2 would both be objects within the filesystem class.
Within a class, all objects must have unique names. If all objects in the cluster have unique names, you can abbreviate some commands by omiting the class name. However, if two or more objects in the cluster database have the same name, you must specify the class in order to uniquely identify the object.
The command syntax is:
command [[class:]object] [attributes] |
where class can be one of the following:
| autoconf |
| cluster |
| cmsd |
| fence |
| failover_net |
| filesystem |
| log |
| node |
| poolnode |
| switch |
where attributes takes a number of formats depending on the context, such as the following:
attribute attribute=value attribute=value1,value2,value3... |
The actual syntax components for any given command vary, based on the needs of the command. For example, the following command requires no parameters to see a summary of the cluster:
cxfs_admin:mycluster> show |
If an object name is unique within the cluster database, you can omit its class name. For example, if the name nodeA is unique within the database:
cxfs_admin:mycluster> show nodeA |
However, if there were multiple objects named production , you must enter the class name:
cxfs_admin:mycluster> show node:production |
Classes and objects may include the following shell-style wildcard characters:
* ? [...] |
| Note: You can use the asterisk (*) alone with
the show and config commands in
place of the object to apply the command to
the entire cluster. if you do not specify any attributes, you can even
omit the * character.
For commands that will change the cluster database, however, SGI recommends that you also include at least one other character in the object's name to ensure that you act upon only the intended objects. For example, if you wanted to unmount both myfileA and yourfileB, you could enter unmount *file*. |
Command names and predefined attribute names are not case-sensitive. However, all other user-supplied attribute values other the node name (in create node nodename) and the switch name (in create switch switchname) are case sensitive.
You can see possible attributes by pressing the <TAB> key after entering the command or object. For example:
cxfs_admin:mycluster> create filesystem <TAB>
Required attributes:
name= : A string
Optional attributes:
forced_unmount= : True/false or enabled/disabled (default is "false")
grio_managed= : True/false or enabled/disabled (default is "false")
mounted= : True/false or enabled/disabled (default is "true")
mountpoint= : A pathname
options= : Nothing, one or more strings (can be empty) |
The required attributes are listed first followed by optional attributes. The list of attributes will differ depending upon the complexity mode; see “Basic and Advanced Mode”.
Partially typing in the attribute name and pressing <TAB> will complete the attribute name (if unique) or show a list of matching attribute names. To see what kind of values are required for an attribute, press <TAB> after the = sign. For example:
cxfs_admin:mycluster> create node os=<TAB> Linux MacOSX Unknown Windows |
Use $ to refer to the object in the last command that you entered at the cxfs_admin prompt. For example, to delete nodeA (if it has a unique name within the cluster database):
cxfs_admin:mycluster> disable nodeA cxfs_admin:mycluster> delete $ |
To specify multiple objects, separate them with a comma. For example:
cxfs_admin:mycluster> show nodeA,nodeB |
You can abbreviate commands, objects, and attributes by entering the first character or two followed by pressing the <TAB> key. If more than one match is available, cxfs_admin shows a list of the possible matches.
Some attributes can take a comma-separated list of values. These attributes also allow you to use the + and - symbols to specify items to add or remove. The symbol must precede the list. For example, for the nodes attribute to the modify command:
nodes=entirelist to specify the entire list of nodes that can mount the filesystem, given as a comma-separated list; you must include the server-capable administration nodes in this list.
nodes=+additionalnodes to specify additional (+) nodes that can mount the filesystem, which will be added to the current list.
nodes=-removednodes to specify which nodes that were previously allowed to mount the filesystem but should now be prevented (-) from mounting the filesystem.
| Note: The +/- symbols are not applicable in prompting mode. |
For an example, see “Create or Modify a CXFS Filesystem with cxfs_admin”.
You can use one of the following methods to set the defaults for the way that cxfs_admin behaves, shown in the order of their precedence:
Use the set command within cxfs_admin:
set editor=emacs|vi (emacs) line_wrap=true|false (true) mode=basic|advanced (basic) stop_on_error=true|false (true)
For example, to change to vi:
cxfs_admin:mycluster> set editor=vi
Usage notes:
editor specifies the editor style ( emacs or vi). The default is emacs .
line_wrap specifies the ability to wrap a line at the edge of the current window (true) or no line wrap (false). The default is true .
mode determines whether prompting and output will show only those values that are required (basic) or all values (advanced). The default is basic. See “Basic and Advanced Mode”.
stop_on_error will abort a command upon encountering an error (true) or keep going ( false). The default is true.
Set the environment variables discussed in Table 11-1.
Table 11-1. cxfs_admin Environment Variables
Environment Variable
Values
CXFS_ADMIN_CLUSTER_NAME
clustername. Setting this value lets you bypass using the -i option if you have multiple clusters connected to the same public network, or want to use cxfs_admin from a node that is not part of the cluster. There is no default.
CXFS_ADMIN_EDITOR
CXFS_ADMIN_LINE_WRAP
CXFS_ADMIN_MODE
CXFS_ADMIN_STOP_ON_ERROR
CXFS_ADMIN_WRITE_MODE
true or false (default), which specifies whether cxfs_admin will attempt to obtain the lock (enter write mode) by default (similar to entering -A on the cxfs_admin command line). To obtain the lock, the host on which cxfs_admin is executed must also have admin access permission and you must be logged in as root. (See “Setting cxfs_admin Access Permissions”.)
Use the .cxfs_admin file in your home directory (as defined by the HOME environment variable) to set any of the following:
cluster_name=clustername editor=emacs|vi line_wrap=true|false mode=basic|advanced stop_on_error=true|false write_mode=true|false
Lines within the .cxfs_admin file that begin with the # character or a space are ignored, as are lines that do not contain the = character.
For example, to use the mycluster cluster in advanced mode and the vi editor by default, include the following in your (or root) $HOME/.cxfs_admin file:
# My settings for cxfs_admin: cluster=mycluster mode=advanced editor=vi
This section discusses the following safety features of cxfs_admin:
By default, cxfs_admin executes in read-only mode, in which the cxfs_admin lock is not set. The lock permits only one user at a time can make changes via cxfs_admin.
 | Caution: The cxfs_admin lock does not prevent other users from using the CXFS GUI while cxfs_admin is running. You should make database changes with only one instance of the CXFS GUI or cxfs_admin command at any given time. |
In read-only mode, you can view the cluster status but you cannot make changes. The cxfs_admin prompt identifies that you are in read-only mode.
To make changes to the cluster database, you must be logged in as root to a host that has admin access permission (see “Setting cxfs_admin Access Permissions”) and be in write mode. Do one of the following as root when cxfs_admin is connected to a node with admin access:
Use the -A option on the cxfs_admin command line to override the default setting for this instance of the cxfs_admin process.
Use the .cxfs_admin file or the CXFS_ADMIN_WRITE_MODE environment variable to permanently override the default setting (making it unnecessary to enter -A on the command line). See “Setting cxfs_admin Defaults”.
Invoke cxfs_admin as normal but then request the lock:
cxfs_admin:mycluster> lock
If you request the lock but it is already held by another user, you will enter in read-only mode. To forcefully obtain the lock from someone else, you can use the steal attribute with the lock command. For example:
cxfs_admin:mycluster (read only) > lock Event at [ Dec 21 15:58:02 ] The administration lock is already held by root@node2 (pid=48449) cxfs_admin:mycluster (read only) > lock steal=true Event at [ Dec 21 15:58:03 ] The administration lock has been stolen from root@node2 (pid=48449) cxfs_admin:mycluster> |
If someone holds the lock while you are using cxfs_admin but later drops it, you can obtain the lock by entering lock (there is no need to steal the lock at this point).
The access command lets you specify hosts that have permission to modify the cluster configuration and hosts that have permission to monitor the cluster state:
access
allow=hostname_or_IPaddress_list
permission=admin|monitor (monitor, only available with allow)
deny=server_name |
By default, all server-capable administration nodes in the cluster are granted admin access (without using the access command).
For example, to grant remotehostA and remotehostB permission to modify the cluster configuration:
cxfs_admin:mycluster> access allow=remotehostA,remotehostB permission=admin |
To grant read-only rights in order to monitor to the cluster configuration and status (monitor is the default access level):
cxfs_admin:mycluster> access allow=remotehostA |
To revoke all access to the cluster database for a host that was previously granted some level of access, use the following command:
cxfs_admin:mycluster> access deny=remotehostA,remotehostB |
To view the current access rights, use the following command:
show access |
For example:
cxfs_admin:mycluster> show access
Event at [ Dec 21 15:58:02 ]
access:
admin=server
monitor=cluster |
Usage notes:
allow specifies the hosts to be granted the specified permission. These hosts must be on the same private network as the cluster nodes. To specify multiple hosts, use a comma-separated list. There are three reserved hostnames:
cluster denotes any node defined in the cluster
server denotes any server-capable administration node, even one that is disabled from CXFS membership (see “Disable a Node with cxfs_admin”)
any denotes any system that is on the private network
permission specifies:
admin provides the ability to view and change the cluster database and perform administrative actions (if cxfs_admin was started in write mode and you are logged in as root). This is the default for all server-capable administration nodes.
monitor provides the ability only to view the current status of the cluster and the current definitions in the cluster database. This is the default for all client-only nodes.
deny specifies the hosts to be denied all access to the cluster database (for hosts that were previously granted some level of access). To specify multiple hosts, use a comma-separated list. The same reserved hostnames as allow apply.
If you have multiple clusters connected to the same public network, use the -i option to identify the cluster name:
cxfs_admin -i clustername |
For example, for the cluster named mycluster:
server-admin# /usr/cluster/bin/cxfs_admin -i mycluster |
On each client-only node, add -i clustername to the cxfs_client.options file. For more information about the cxfs_client.options file, see CXFS 7 Client-Only Guide for SGI InfiniteStorage.
You can also use the CXFS_ADMIN_CLUSTER_NAME environment variable. See “Setting cxfs_admin Defaults”.
At any time, you can enter help or ? to see help text.
To see help for a given topic:
help topicname |
The list of general topics includes the following:
attributes commandline commands cxfs objects overview setup syntax tasks waiting |
There is also help for each cxfs_admin command. For example, to see help about the create command:
cxfs_admin:mycluster> help create |
To see all of the available help topics, press the <TAB> key:
cxfs_admin:mycluster> help <TAB> |
To see a list of available commands for an object, such as a class like filesystem or a specific instance of a class like the filesystem myfs, use the ops (operations) command:
ops object |
For example:
cxfs_admin:mycluster> ops filesystem
Event at [ Dec 21 15:58:02 ]
Commands for "filesystem":
create, ops, show
cxfs_admin:mycluster> ops myfs
Event at [ Dec 21 15:58:03 ]
Commands for "filesystem:myfs":
delete, modify, mount, ops, show, unmount |
You can also use the clconf_info tool to view status. See Chapter 14, “Monitoring Status”.
For help with error messages, see “cxfs_admin Errors” in Chapter 15.
The cxfs_admin command operates in two complexity modes:
Basic mode, which:
Shows only the required options and attributes in show output
Provides a list of possible choices when using the <TAB> key
Uses prompting for only the required fields
Advanced mode, which:
Provides a list of possible choices when using the <TAB> key
Prompts for all possible fields, displays all attributes
Includes debugging information in output
Note: You should only use the advanced-mode commands and attributes at the advice of SGI support. Using the advanced-mode commands or changing advanced-mode attributes may induce unexpected behavior.
Advanced-mode commands and attributes are not included in prompts or <TAB> key completion when you are in basic mode. However, you can still manually enter an advanced attribute if you know it, even in basic mode. The advanced-mode commands and attributes are noted in their help topics.
You can enter advanced mode by using cxfs_admin -a on the command line or by entering the following cxfs_admin command:
cxfs_admin:mycluster> set mode=advanced |
To return to basic mode:
cxfs_admin:mycluster> set mode=basic |
For example, the following output shows only the basic-mode information for the node named cxfsxe5:
cxfs_admin:clusterOne> set mode=basic
Event at [ Dec 20 12:32:44 ]
Mode is set to basic
cxfs_admin:clusterOne> show cxfsxe5
Event at [ Dec 20 12:32:54 ]
node:cxfsxe5:
cellid=1
enabled=true
os=Linux
private_net:
10.0.199.1
status:
age=23
connected=true
fencing=Stable
license:
have_license=true
summary=Stable
version=7.0.0.3
wwns:
210000e08b123d95
type=server_admin |
| Note: The have_license field reports licenses for client-only nodes that have been obtained from the server. It does not display information about the server licenses. For more information about licensing, see “Show License Information with cxfs_admin”. |
For example, the following output shows all information that is available in advanced mode for cxfsxe5:
cxfs_admin:clusterOne > set mode=advanced
Event at [ Dec 20 12:33:48 ]
Mode is set to advanced
cxfs_admin:clusterOne> show cxfsxe5
Event at [ Dec 20 12:33:53 ]
node:cxfsxe5:
autoconf=false
cellid=1
clustername=clusterOne
enabled=true
failpolicy=Fence,Shutdown
hostname=cxfsxe5.americas.sgi.com
nodeid=1
os=Linux
private_net:
10.0.199.1
status:
age=23
build=07:39:08 Sep 22 2009
connected=true
fencing=Stable
license:
have_license=true
member=true
stable=true
summary=Stable
version=7.0.0.3
wwns:
210000e08b123d95
type=server_admin |
| Note: If a client is not connected to the cluster, the build and version fields will not display because the node cannot respond to for requests for this information. |
Some cxfs_admin commands will prompt you for required attributes if you press ENTER after the command name. To see information about the legal values for an attribute, press <TAB> after a question.
For example:
cxfs_admin:clusterOne> create <RETURN>
What do you want to create? The following can be used:
autoconf, cluster, failover_net, filesystem, node, switch |
In basic mode, you are only prompted for required parameters. To be prompted for all possible parameters, use advanced mode. See “Basic and Advanced Mode”.
Depending upon the context, cxfs_admin prompts will vary based upon your answers to previous prompts. For example, if you specify that a node's type value is client-only , cxfs_admin will prompt you for the operating system.
To exit from prompt mode, send an interrupt signal (typically, press Ctrl-C).
The history command displays a list of commands that have been used in cxfs_admin since it was started:
Display all of the commands (up to the previous 1000 commands):
history
Limit the commands to the last specified number of items:
history num=number_of_items
For example, to display only the last 10 commands:
cxfs_admin:mycluster> history num=10
Clear the history:
history clear
Send the history to a file (full pathname or relative pathname):
history output=pathname
For example, to send the history output to the file /tmp/myhistory :
cxfs_admin:mycluster> history output=/tmp/myhistory
Some commands in cxfs_admin take a noticeable period of time to complete. cxfs_admin displays informational updates as a command progresses or a period character if nothing has changed within 2 seconds.
After 1 minute without change, a command will terminate. This may happen when there is a problem in creating or modifying a node or filesystem. The update message shows the problem status.
To interrupt a command, send an interrupt signal (usually Ctrl-C).
You can enter cxfs_admin commands directly from the cxfs_admin command line by using the -c option, as follows:
server-admin# /usr/cluster/bin/cxfs_admin -c "cxfs_admin_commands" [-i clustername] |
You can execute a series of cxfs_admin commands by using the -f option and specifying an input file:
server-admin# /usr/cluster/bin/cxfs_admin -f command_file [-i clustername] |
For example, suppose the file /tmp/showme contains the following:
cxfs6# more /tmp/showme show access set |
You can execute the following command, which will yield the indicated output (line breaks shown for readability):
# /usr/cluster/bin/cxfs_admin -f /tmp/showme
Connecting to the CXFS server for the "clusterOne" cluster...
Event at [ Dec 20 12:49:17 ]
access:
admin=server
monitor=cluster
Event at [ Dec 20 12:49:17 ]
mode=basic
editor=emacs
line_wrap=false
stop_on_error=true |
| Note: To make changes, you must execute cxfs_admin in write mode, such as by using the -A option. See “Making Changes Requires Both Write Mode and admin Permission”. |
This section discusses the following:
| Note: The entire cluster status information is sent to each server-capable administration node each time a change is made to the cluster database; therefore, the more server-capable administration nodes in a configuration, the longer it will take. |
The autoconf class creates a rule that determines whether the specified new client-only nodes can be automatically configured into the CXFS cluster database. This is known as easy client configuration.
| Note: The autoconf creation action only applies
to hosts that are not currently defined in the cluster database.
Before deleting a node created by autoconf, you must set the autoconf policy to disallowed . If the policy remains in allowed mode, the node will be automatically re-created after the node is deleted. |
You can specify a single hostname and/or a range of IP addresses to be allowed (whitelisted) or disallowed ( blacklisted). Only those nodes specified in a whitelist and not specified in a blacklist can be automatically configured. (That is, the blacklist has higher priority than the whitelist.)
Each autoconf rule must have a unique name. By default, only nodes that are defined by an autoconf rule set to enable_node=true will be defined as enabled and will be allowed to join the cluster. (If you set enable_node=false , the node will be defined in the database but disabled/inactive.)
To create a new rule (line breaks shown here for readability):
create autoconf rule_name=rulename policy=allowed|disallowed (allowed) hostname=hostname startIP=startingIPaddress endIP=endingIPaddress enable_node=true|false (true)
To change an existing rule:
modify [allowed:|disallowed:]rulename
To delete an existing rule:
delete [allowed:|disallowed:]rulename
To show an existing rule:
show [allowed:|disallowed:]rulename
To show all existing rules:
show autoconf
To show all allowed rules, which displays the rules for client-only nodes that are allowed to be automatically configured (whitelisted):
show allowed
To show all disallowed rules, which displays the rules for client-only nodes that are explicitly not allowed to be automatically configured (blacklisted):
show disallowed
For example, suppose you want nodes NodeA and NodeB to be automatically configured and enabled in the cluster. You would create two rules:
cxfs_admin:mycluster> create autoconf rule_name=ruleNodeA policy=allowed hostname=NodeA enable_node=true cxfs_admin:mycluster> create autoconf rule_name=ruleNodeB policy=allowed hostname=NodeB enable_node=true |
If you then wanted to change the rule so that NodeA could not be automatically configured, you would modify the autoconf rule named ruleNodeA:
cxfs_admin:mycluster> modify autoconf rule_name=ruleNodeA policy=disallowed hostname=NodeA |
| Note: If NodeA was already defined in the cluster database, this would not affect the existence of that node but would prevent it from being recreated automatically in the future. |
In this case, you could also delete ruleNodeA, because not listing a node in an autoconf allow rule is the same as disallowing it.
Suppose all of your client-only nodes have IP addresses within the range 192.168.0.1 through 192.168.0.20. To let all of the nodes be configured automatically except for the node named bert, you would need to create two rules:
cxfs_admin:mycluster> create autoconf rule_name=allowall policy=allowed startIP=192.168.0.1 endIP=192.168.0.20 enable_node=true cxfs_admin:mycluster> create autoconf rule_name=rulebert policy=disallowed |
If there is a node that is not within the IP address range that you also want to allow, you can include it in the same rule. For example, the following would allow all nodes in the range 192.168.0.1 through 192.168.0.20 plus the node named ernie (that is in a different IP range):
cxfs_admin:mycluster> create autoconf rule_name=allowall policy=allowed startIP=192.168.0.1 endIP=192.168.0.20 hostname=ernie enable_node=true Event at [ Dec 20 12:53:50 ] "allowall" is created. After creating all desired autoconf rules, unlock all cxfs_admin sessions to allow the defined nodes to then be automatically configured. |
After you have finished creating or modifying all of the desired autoconf rules, you must unlock all cxfs_admin sessions in order for nodes to be automatically configured. (The automatic configuration process must have access to the cxfs_admin lock.) If the node that is created or reconfigured via autoconf is unable to join cluster membership, you must reboot the node.
When you create a new node, you can use the failover_net attribute to specify the preferred selection order of the private IP addresses (private_net), which is particularly important in an IPv6 environment. See:
| Note: Before adding a server-capable administration node, you must
first unmount any filesystems for which the node will be a potential metadata
server.
You should create all server-capable administration nodes before creating any client-only nodes to ensure that the server-capable administration nodes get the lowest cell IDs. The cell ID is assigned by CXFS according to the order in which the nodes are created. |
To define a node, use the following command and attributes (line breaks shown here for readability, defaults in parentheses):
create node
name=nodename
type=client_only|server_admin (client_only)
os=Linux|MacOSX|Windows|Unknown (Linux)
private_net=private_network_IPaddress_list|hostname_list
Advanced-mode:
enabled=true|false (true)
hostname=hostname (FQDN_of_nodename)
nodeid=nodeID (assigned by cxfs_admin)
partition_id=partition_number
failpolicy=FenceReset,Fence,Reset,Shutdown (Fence,Shutdown)
reset_method=nmi|powerCycle|reset (reset)
reset_port= bmc|l2
reset_password=password (password)
reset_user=username (admin)
reset_status=enabled|disabled (enabled)
reset_node=server_admin_node_sending_reset_command
reset_comms=ipmi|network|tty
reset_device=port|IP_address_or_hostname_of_device |
You will use a similar set of subcommands to modify a node via the modify command. (To modify a node, you must first disable it.)
When adding the first server-capable administration node, you must restart it or restart CXFS services and cluster services on the node:
server-admin# service cxfs stop server-admin# service cxfs_cluster stop server-admin# service cxfs_cluster start server-admin# service cxfs start |
When you create additional nodes, they will automatically be enabled and join the cluster.
To use prompting mode, press <ENTER>. To obtain information about legal values, press <TAB>.
For example, to create a client-only node, you could do the following, pressing the <TAB> key to see the list of operating system values:
cxfs_admin:mycluster> create node Specify the attributes for create node: name? newnode type? client_only os? <TAB> Linux MacOSX Unknown Windows os? macosx private_net? 192.168.0.178 Event at [ Dec 21 15:58:02 ] Node "newnode" has been created, waiting for it to join the cluster... Waiting for node newnode, current status: Inactive Waiting for node newnode, current status: Establishing membership Waiting for node newnode, current status: Probing XVM volumes Operation completed successfully |
To create a server-capable administration node using the defaults, you must delete the client_only default for type and enter server_admin. For example:
cxfs_admin:mycluster> create node Specify the attributes for create node: name? newnode type? server_admin private_net? 192.168.0.178 Event at [ Dec 21 15:58:02 ] Node "newnode" has been created, waiting for it to join the cluster... Please restart all cxfs and cluster services on the server "newnode" to make it join the cluster. |
To create a server-capable administration node in advanced mode, which can prompt you to set additional values (such as for reset_method and failpolicy):
cxfs_admin:mycluster> set mode=advanced cxfs_admin:mycluster> create node Specify the attributes for create node: name? newnode type? server_admin private_net? 192.168.0.178 enabled? true failpolicy? Reset,Shutdown hostname? newnode.example.com nodeid? 1 partition_id? reset_method? reset reset_port? bmc reset_password? reset_user? reset_status? enabled reset_node? mds2 reset_comms? ipmi reset_device? newnode-bmc.mycompany.com Event at [ Dec 21 15:58:02 ] Node "newnode" has been created, waiting for it to join the cluster... Please restart all cxfs and cluster services on the server "newnode" to make it join the cluster. |
You will use a similar set of subcommands to modify a node via the modify command. (To modify a node, you must first disable it.)
To modify the failpolicy to eliminate Shutdown:
cxfs_admin:mycluster> modify newnode failpolicy=Reset,Fence |
Basic-mode usage notes:
name is a simple hostname (such as lilly) or a fully qualified domain name (such as lilly.example.com ) or an entirely different name (such as node1). It cannot begin with a number or an underscore (_), or include any whitespace characters, and can be at most 255 characters.
type specifies the function of the node. Enter one of the following:
client_only is a node that shares CXFS filesystems but will never be a CXFS metadata server. Most nodes should be client-only nodes.
server_admin is a Linux node that is a potential CXFS metadata server. (You will use the create filesystem command to define the specific filesystem for which this node can be a metadata server.)
os is one of the following for client-only nodes:
Linux MacOSX Unknown Windows private_net is the IP address or hostname of the private network. (The hostname must be resolved in the /etc/hosts file.) SGI requires that this network be private; see “Private Network” in Chapter 1.
There can be up to 8 network interfaces. The order in which you specify the interfaces determines their priority; the first interface will be priority 1, the second priority 2, and so on. There is no default. Unless you specify the failover_net attribute, the interfaces must be specified in the same order according to subnet on each node in the cluster.
To limit the networks chosen from this list or reprioritize the list, use the failover_net attribute. See “Network Failover Modification Tasks with cxfs_admin”.
For more information about using the hostname, see “Hostname Resolution and Network Configuration Rules” in Chapter 6.
You can use private_net for networks using both IPv6 and IPv4. For IPv6, the address must be a link-local address. For example:
cxfs_admin:mycluster> create node name=mynode private_net=192.168.0.5,fe80::204:23ff:fee2:d861
Advanced-mode usage notes:
enabled determines if a node will be able to obtain CXFS membership (true) or not ( false). By default, the new node is enabled (true). To enable a command created with enabled=false, use the enable command. See “Enable a Node with cxfs_admin”.
failpolicy determines what happens to a failed node. You can specify up to three methods. The second method will be completed only if the first method fails; the third method will be completed only if both the first and second options fail. Separate options by commas (not whitespace). The option choices are as follows:
Fence disables access to the SAN from the problem node. Fencing provides faster recovery of the CXFS kernel membership than reset.
FenceReset performs a fence and then, if the node is successfully fenced, also performs an asynchronous reset of the node via a system controller; recovery begins without waiting for reset acknowledgement.
Note: SGI recommends that a server-capable administration node include Reset in its failpolicy (unless it is the only server-capable administration node in the cluster). See “Data Integrity Protection” in Chapter 1. The FenceReset and Fence policies are mutually exclusive.
Reset performs a system reset via a system controller. This action requires a reset_method value; see “Hardware and Software Requirements for Server-Capable Administration Nodes” in Chapter 1.
Shutdown tells the other nodes in the cluster to wait for a period of time (long enough for the node to shut itself down) before reforming the CXFS kernel membership. (However, there is no notification that the node's shutdown has actually taken place.)
Caution: Because there is no notification that a shutdown has occurred, if you have a cluster with no tiebreaker, you must not use the Shutdown setting for any server-capable administration node in order to avoid split clusters being formed. See “Node Shutdown” in Chapter 2. You should not use the Shutdown fail policy on client nodes if you choose dynamic CXFS kernel heartbeat monitoring for the cluster.
Note: If the failure hierarchy contains Reset or FenceReset, the reset might be performed before the system kernel core-dump can complete, resulting in an incomplete core-dump. For a list of valid failpolicy sets, see “Data Integrity Protection” in Chapter 1.
For example, to perform a reset only if a fencing action fails, specify the following:
failpolicy=Fence,Reset
Note: If the fail policy does not include Shutdown and all of the other actions fail, CXFS will stall membership until the failed node either attempts to join the cluster again or until the administrator intervenes by using cms_intervene. Objects held by the failed node stall until membership finally transitions and initiates recovery. For more information, see the cms_intervene(8) man page. To perform a fence and an asynchronous reset, specify the following:
failpolicy=FenceReset
hostname is the fully qualified domain name. Use the ping to display the fully qualified hostname. Do not enter an IP address. The default for hostname is the fully qualified domain name for the value of name.
nodeid is an integer in the range 1 through 32767 that is unique among the nodes in the cluster. If you change this value after a node has been defined, you must reboot the affected node. You do not normally need to specify this attribute because cxfs_admin will calculate an ID for you.
partition_id uniquely defines a partition in an Altix 3000 series system or Altix 4700 system. For a non-partitioned system, this attribute is not required (the default unassigned). To unset the partition ID, use a value of 0. You can find the partition ID by reading the proc file. For example, for an Altix 3000:
altix# cat /proc/sgi_sn/partition_id 0
The 0 indicates that the system is not partitioned. If the system is partitioned, the number of partitions (such as 1, 2, etc.) is displayed.
reset_method specifies the action that will automatically be taken by CXFS if reset_status=enabled. It can be one of the following:
powerCycle shuts off power to the node and then restarts it
Note: If you specify reset_comms=ipmi , reset_method=powerCycle and reset_method=reset have the same effect, which is to perform a hard reset (that is, perform a power reset with ipmitool).
reset simulates the pressing of the reset button on the front of the machine (recommended)
nmi (nonmaskable interrupt) performs a core-dump of the operating system kernel, which may be useful when debugging a faulty machine
Note: NMI depends upon kernel support, which may not be present on all SGI ia64 systems; if the kernel support is not provided, the nmi setting will not work.
NMI is not available on systems containing a baseboard management controller (BMC), such as SGI x86_64 systems.
The default is reset.
reset_port is the system controller port type based on the node hardware, as shown in Table 11-2.
reset_password is the password for the node's system controller port (not the node's root password or PROM password). On some nodes, the system administrator may not have set this password. If you wish to set or change the system controller password, consult the hardware manual for your node. The default is password.
reset_user is the user name for the node's system controller port, which might not apply to all systems. If you wish to set or change the system controller user, consult the hardware manual for your node. The default is admin.
reset_status specifies if the automatic system reset capability is turned on (enabled) or turned off (disabled). Using disabled lets you provide information about the system controller but temporarily disable reset (meaning that CXFS cannot reset the node). The default for nodes with system controllers is enabled, the default for nodes without system controllers is disabled; see “Hardware and Software Requirements for Server-Capable Administration Nodes” in Chapter 1.
reset_node specifies the name of the server-capable administration node (the owner node) that will send the reset command to the target node. It can be the logical name, hostname, or fully qualified domain name. If you use reset_comms=tty , serial cables must physically connect the target node and the owner node through the system controller port. The owner node must be defined in the cluster database.
reset_comms is tty for TTY serial devices, network for network reset to systems with L2 system controllers (you can use network reset even if you have hardware reset capability), or ipmi for intelligent platform management interface (IPMI) network reset to systems with BMC system controllers.
reset_device is one of the following:
For systems with serial ports (reset_comms=tty ), this is the name of the terminal port (TTY) on the owner node (the node issuing the reset). A serial cable connects the terminal port on the owner node to the system controller of the node being reset. /dev/ttyd2 is the most commonly used port, except on Altix 350 systems (where /dev/ttyIOC0 is commonly used).
Note: Check the owner node's specific hardware configuration to verify which TTY device to use. For systems with network-attached L2 system controllers (reset_comms=network), this is the IP address or hostname of the L2 controller on the node being reset. For example:
reset_device=nodename-l2.mycompany.com
For systems with network-attached BMC system controllers (reset_comms=ipmi), this is the IP address or hostname of the BMC controller on the node being reset. For example:
reset_device=nodename-bmc.mycompany.com
Altix 3000 Bx2 system:
reset_comms=network reset_device=nodename-l2.mycompany.com |
Altix 350 system without an L2:
reset_comms=tty reset_device=/dev/ttyIOC0 |
SGI x86_64 system with a BMC:
reset_comms=ipmi reset_device=nodename-bmc.mycompany.com |
Table 11-2. System Controller Types
bmc | l2 |
|---|---|
Any SGI x86_64 system | Any SGI ia64 system with an L2 |
| Prism |
The following is an example of a node created for BMC reset, where the default for reset_password is password as explained in Appendix C, “BMC System Controller” (line breaks added here for readability):
cxfs_admin:mycluster> create node name=node1 type=server_admin private_net=192.168.0.168 enabled=true hostname=node1.example.com failpolicy=Fence,Reset,Shutdown nodeid=1 reset_method=reset reset_port=bmc reset_status=enabled reset_node=mds2 reset_comms=ipmi reset_device=node1-bmc.mycompany.com |
| Note: Before deleting a node that was created by autoconf
, you must set the autoconf policy to
disallowed. If the policy remains in allowed
mode, the node will be automatically re-created after the node is deleted.
Before deleting a server-capable administration node, you must first unmount any filesystems on that node and disable the node. |
To delete a node from the cluster and the cluster database, use the following command:
delete [node:]nodename |
If the node is enabled (which is the default), you must disable it before you delete it. For example, if mynode is a unique name in the cluster database:
cxfs_admin:mycluster> disable mynode Node "mynode" has been disabled, waiting for it to leave the cluster... Waiting for node mynode, current status: Disabled and unmounting Operation completed successfully cxfs_admin:mycluster> delete mynode |
| Note: If you delete an active metadata server, cxfs_admin will enter read-only mode. You can use the lock or lock steal=true to reenter lock mode. For more information, see “Making Changes Safely”. |
To allow a disabled node to join the cluster, enter the following:
enable [node:]nodename |
For example, if node1 is a unique name in the cluster database:
cxfs_admin:mycluster> enable node1 |
To prevent a node from joining the cluster, enter the following:
disable [node:]nodename |
For example, if node1 is a unique name in the cluster database:
cxfs_admin:mycluster> disable node1 |
| Note: This procedure is strongly recommended for stopping CXFS services on CXFS server-capable administration nodes or on a client-only tie-breaker node that must be shutdown for maintenance or otherwise taken offline. These types of nodes will affect the CXFS kernel membership quorum calculation as long as they are configured as enabled in the cluster database. |
After you have disabled a node, the node is no longer an active member of the cluster.
| Caution: If you disable a node, it will be marked as Disabled and it will therefore not rejoin the cluster after a reboot. To allow a node to rejoin the cluster, you must enable the node. See “Enable a Node with cxfs_admin”. |
You can display a node's parameters with the following command:
show [node:]nodename |
For example, if cxfsxe5 is a unique name in the cluster database:
cxfs_admin:clusterOne> show cxfsxe5
Event at [ Dec 21 13:35:25 ]
node:cxfsxe5:
cellid=1
enabled=true
os=Linux
private_net:
10.0.199.1
status:
age=44
connected=true
fencing=Stable
license:
have_license=true
summary=Stable
version=7.0.0.3
wwns:
210000e08b123d95
type=server_admin |
| Note: The have_license field reports licenses for client-only nodes that have been obtained from the server. It does not display information about the server licenses. For more information about licensing, see “Show License Information with cxfs_admin”. |
You can see a list of all of the nodes that have been defined with the following command:
show node |
For example (output truncated):
cxfs_admin:clusterOne > show node
Event at [ Dec 21 13:36:02 ]
node:
bert:
cellid=0
enabled=true
os=Linux
private_net:
10.0.199.3
status:
age=33
connected=true
fencing=Stable
license:
have_license=true
summary=Stable
version=7.0.0.3
wwns:
100000062b126ff8, 100000062b126ff9
type=server_admin
cxfsxe5:
cellid=1
enabled=true
os=Linux
private_net:
10.0.199.1
status:
age=44
connected=true
fencing=Stable
license:
have_license=true
summary=Stable
version=7.0.0.3
wwns:
210000e08b123d95
type=server_admin
cxfsxe10:
cellid=3
enabled=false
os=Linux
private_net:
10.0.199.4
status:
age=-
connected=false
summary=Disabled
type=client_only
penguin17:
cellid=4
enabled=true
os=Linux
private_net:
10.0.199.17
status:
age=54
client=stable
connected=true
fencing=Stable
filesystems=up
license:
allocated=1
have_license=true
oem=none
os=SGI_Software_Foundation
membership=up
summary=Stable
version=7.0.0.3
wwns:
210000e08b1a07d8
xvm=up
type=client_only
... |
See also “Displaying the Keys with cxfs_admin After Installing CXFS” in Chapter 5.
The cmsd daemon controls CXFS kernel membership and heartbeat.
You must use the following command to allow CXFS kernel membership for the local server-capable administration node (the node on which cxfs_adminis running) after fixing the problems that required a forced CXFS shutdown (see “Shutdown of the Database and CXFS” in Chapter 12):
cxfs_admin:mycluster> enable cmsd |
You should revoke CXFS kernel membership of the local node only if an error requires you to perform a forced CXFS shutdown:
cxfs_admin:mycluster> disable cmsd |
The result of the disable cmsd command is considered as a node failure by the rest of the cluster. The cluster may then fail due to a loss of CXFS kernel membership quorum or it may reset the failed node. To avoid the reset, modify the local node's definition to disable the system controller status.
| Note: The enable cmsd and disable cmsd commands are only effective when cxfs_admin is executed directly on a server-capable administration node. They only affect the local node (the node on which cxfs_admin is running). |
| Note: The control command must be run when cxfs_admin is executed directly on a server-capable administration node. |
You can use the control command from a server-capable administration node to manually control and contact a target node that has a system controller. The target node must have the following attributes defined in the cluster database (see “Create or Modify a Node with cxfs_admin”):
reset_port reset_node reset_comms reset_device |
The node may also have the following attributes defined in the cluster database:
reset_password reset_user |
You can do the following:
SGI recommends the reset method for system control on server-capable administration nodes. To reset the target node, which simulates the pressing of the reset button on the front of the machine (recommended), enter the following:
cxfs_admin:mycluster> control target-nodename operation=reset |
For example, to reset the node named mds1:
cxfs_admin:mycluster> control mds1 operation=reset |
When the target node is reset, other nodes in the cluster will detect the change and remove the target node from the active cluster. When the target node reboots, it will rejoin the CXFS kernel membership.
To shut off power to the target node and then restart it, enter the following:
cxfs_admin:mycluster> control target-nodename operation=powerCycle |
To send a nonmaskable interrupt to the target node, which performs a core-dump of the operating system kernel that may be useful when debugging a faulty machine, enter the following:
cxfs_admin:mycluster> control target-nodename operation=nmi |
| Note: NMI depends upon kernel support, which may not be present
on all SGI ia64 systems; if the kernel support is not provided, the
nmi setting will not work.
NMI is not available on systems containing a baseboard management controller (BMC), such as SGI x86_64 systems. |
| Note: Prompting for the detach and attach commands is available only in advanced mode. |
The detach command does the following:
Removes the node from the cluster definition
Releases the node's cell ID
Moves the node to the pool (making it part of the poolnode class)
The attach command does the following:
Adds the node to the cluster definition
Creates a new cell ID for the node
Removes the node from the poolnode class
| Note: Normally, you should only use the detach
and attach commands if you want to release/establish
a node's cell ID.
If you want to change a node's cluster membership. disable and enable commands. (A disabled node still keeps its cell ID.) See: |
To detach a node, enter the following:
detach nodename |
To return the node to the cluster, enter the following:
attach nodename |
See also:
This section discusses the following:
To create the cluster, use the following command (line breaks shown here for readability, defaults in parentheses):
create cluster
name=clustername
Advanced-mode:
id=clusterID
heartbeat_monitor=dynamic|static (static) |
For example:
cxfs_admin:> create cluster name=mycluster |
Basic-mode usage notes:
clustername specifies the logical name of the cluster. It cannot begin with a number or an underscore (_), or include any whitespace characters, and can be at most 255 characters.
Advanced-mode usage notes:
id is a unique number within your network in the range 1 through 255. The default is 1The cluster ID is used by the operating system kernel to make sure that it does not accept cluster information from any other cluster that may be on the network. The kernel does not use the database for communication, so it requires the cluster ID in order to verify cluster communications. This information in the kernel cannot be changed after it has been initialized; therefore, you cannot change a cluster ID after the cluster has been defined.
Note: Clusters must have unique IDs. See “Use a Private Network and Unique Cluster Name/ID” in Chapter 2. heartbeat_monitor specifies how CXFS kernel membership is monitored. All nodes send CXFS kernel heartbeat messages once per second. If a node does not receive a heartbeat within a defined period, that node loses membership and is denied access to the cluster's filesystems. The defined period is one of the following:
static: Monitors constantly at 1-second intervals and declares a timeout after 5 consecutive missed seconds (default).
dynamic: Starts monitoring only when the node is processing a message from another node (such as for token recall or XVM multicast) or when the client monitors the server because it has a message pending (for example, a token acquire or metadata operation). Once monitoring initiates, it monitors at 1-second intervals and declares a timeout after 5 consecutive missed seconds, just like static monitoring. Dynamic heartbeat monitoring is appropriate for clusters that have clients with heavy workloads; using it avoids inappropriate loss of membership. However, it may take longer to recover a client's tokens and other state information when there is an actual problem.
Note: You should not use the Shutdown fail policy on client nodes if you choose dynamic heartbeat monitoring for the cluster.
To modify the cluster, use the following command (line breaks shown here for readability, defaults in parentheses):
modify clustername
tiebreaker=client_only_nodename
notify_addr=[+/-]email_addresses
notify_cmd=email_program
Advanced-mode:
heartbeat_monitor=dynamic|static (static)
|
| Note: You cannot change the cluster's name or ID. |
For example, if mycluster is a unique name in the cluster database, to make the client-only node clientA the CXFS tiebreaker:
cxfs_admin:mycluster> modify mycluster tiebreaker=clientA |
To email status changes to the users joe and sally using the /usr/bin/mail program, enter the following (line breaks shown here for readability):
cxfs_admin:mycluster> modify mycluster notify_cmd=/usr/bin/mail [email protected],[email protected] |
Basic-mode usage notes:
clustername identifies the object for which values will be modified (you cannot change the name of the cluster).
tiebreaker specifies the CXFS tiebreaker. See “Specify a Tiebreaker with cxfs_admin”.
notify_addr specifies the email address that CXFS will contact when there are cluster and node status changes. To specify multiple addresses, separate them with commas.
notify_addr specifies the email program to use, such as /usr/bin/mail.
Advanced-mode usage notes:
heartbeat_monitor specifies how CXFS kernel membership is monitored. See the description in “Create a Cluster with cxfs_admin”.
The CXFS tiebreaker node determines whether a CXFS kernel membership quorum is maintained when exactly half of the server-capable administration nodes can communicate with each other. There is no default CXFS tiebreaker.
| Caution: SGI recommends that you use client-only nodes as tiebreakers
to ensure that the cluster remains operational. cxfs_admin
will only let you specify a server-capable administration node as a tiebreaker
if the cluster contains four or more server-capable administration nodes
and an even number of server-capable administration nodes.
To ensure data integrity for all nodes, you must use reset or I/O fencing. Clusters should have an odd number of server-capable administration nodes. If you have an even number of server-capable administration nodes, define a CXFS tiebreaker node. |
To set the CXFS tiebreaker node, use the modify command as follows:
modify [cluster:]clustername tiebreaker=client_nodename |
For example:
cxfs_admin:mycluster> modify mycluster tiebreaker=myclient |
To unset the CXFS tiebreaker node, do not supply a value for tiebreaker. For example:
cxfs_admin:mycluster> modify mycluster tiebreaker= |
To delete an inactive cluster, use the following command:
delete [cluster:]clustername |
For example, if mycluster is a unique name in the cluster database:
cxfs_admin:mycluster> delete mycluster |
However, you cannot delete an active cluster; you must first unmount and delete the filesystems, disable and delete the nodes, and so on.
To display the cluster, use the following command:
show cluster |
For example:
cxfs_admin:clusterOne > show cluster
Event at [ Dec 21 14:08:26 ]
cxfs:cluster:
clusterOne:
access:
admin=server
monitor=cluster
autoconf:
allowed:
pg-27
disallowed:
(none)
clusterid=9
failover_net:
(none)
filesystem:
zj01s0, zj01s1
heartbeat_monitor=static
log:
clconfd, cli, crsd, diags
node:
bert, cxfsxe5, cxfsxe10, penguin17, pg-27
status:
filesystems:
stable=true
summary=Stable
licenses:
cxfs_client
nodes:
stable=true
summary=Stable
stable=true
summary=stable
switch:
brocade26cp1
tiebreaker= |
To show the number of CXFS licenses available for the cluster, use the following command:
show licenses |
For example:
cxfs_admin:clusterOne > show licenses
Event at [ Dec 21 16:07:25 ]
status:licenses:
cxfs_client:
allocated=1:
valid=20 |
To show the version of CXFS software that is running on each node defined in the cluster, use the following command:
show version |
For example:
cxfs_admin:clusterOne > show version Event at [ Dec 22 10:57:19 ] node:bert:status:version=7.0.0.3 node:cxfsxe5:status:version=7.02.0.3 node:cxfsxe10:status:version=7.0.0.3 node:penguin17:status:version=7.0.0.3 node:penguin19:status:version=7.0.0.3 node:pg-27:status:version=7.0.0.3 |
The filesystem class represents the clustered XVM volumes that can be mounted by CXFS nodes. Before you can create a filesystem definition, you must create the clustered XVM volume and make the filesystem with mkfs.
By default, the filesystem:
Uses the XVM device of the same name
Enables all nodes to mount the filesystem
Mounts the filesystem in /mnt/
Is not managed by GRIOv2
To override these defaults, use the optional attributes listed below.
This section discusses the following:
Use the following commands to define a filesystem and the nodes on which it may be mounted (line breaks shown here for readability, defaults in parentheses):
create filesystem
name=fsname
options=mount_options
forced_unmount=true|false (false)
mountpoint=mountpoint (/mnt/fsname)
mounted=true|false (true)
grio_managed=true|false (false)
grio_qual_bandwidth=qualified_bandwidth
Advanced-mode:
device=volumename (fsname)
servers=server_list (all servers are potential MDS)
nodes=nodelist (all nodes can mount)
mount_new_nodes=true|false (true)
|
You will use a similar set of subcommands to modify a CXFS filesystem via the modify command. To modify the following attributes, you must first unmount the filesystem:
| device |
| grio_qual_bandwidth |
| mountpoint |
| options |
| Note: You must create XVM volumes with xvm and
mkfs before you set it up using cxfs_admin.
The device value must be the XVM volume name. In basic mode, you are not prompted for a device name; cxfs_admin uses the value for name and prepends /dev/cxvm/ to it. Therefore, in basic mode, the value you specify for name must be the XVM volume name. If you want to use a name value other than the XVM volume name, then you must use advanced mode and specify the device attribute. |
Basic-mode usage notes:
name specifies the name of the filesystem:
If you also specify a value for device, then name can be any string that does not begin with a number or an underscore (_), or include any whitespace characters, and can be at most 255 characters. For example, if the full XVM volume name is /dev/cxvm/concat1:
cxfs_admin:mycluster> create filesystem name=filesys1 device=concat1
Note: A filesystem name that is longer than 18 characters will be truncated in the status output. For more information about this command, see “cxfs_admin and Status” in Chapter 14. If you do not specify a value for device, then name must be the name of the XVM volume name following /dev/cxvm. For example:
cxfs_admin:mycluster> create filesystem name=concat1
options specifies the mount options that are passed to the mount operating system command. These mount options control access to the specified filesystem. For a list of supported mount options, see the CXFS 7 Client-Only Guide for SGI InfiniteStorage. By default, this is unassigned.
Specify multiple mount options as a comma-separated list. For example, the following specifies that the myfs filesystem uses inode64 allocation and does not update the access time stamps for files and directories:
cxfs_admin:mycluster> create filesystem name=myfs options=inode64,noatime
Note: No validation is done on the mount options in cxfs_admin , so an invalid option may prevent the filesystem mounting on all nodes. forced_unmount controls the action that CXFS takes if there are processes that have open files or directories in the filesystem to be unmounted:
If set to true, the processes will be killed and the unmount will occur
If set to false, the processes will not be killed and the filesystem will unmount only after all references to the filesystem have been closed (default)
mounted specifies whether a new filesystem is mounted on all nodes in the cluster (true) or not mounted on any nodes (false). By default, the new filesystem is mounted on all nodes (true).
mountpoint specifies a mount point for the filesystem. The mount point is a directory to which the XVM volume is attached. This directory name must begin with a slash (/). The default is /mnt/fsname.
For example, to create a filesystem named myfs and use the default mount point of /mnt/myfs:
cxfs_admin:mycluster> create filesystem name=myfs
To create the myfs filesystem but use a mount point of /tmp/myfs:
cxfs_admin:mycluster> create filesystem name=myfs mountpoint=/tmp/myfs
grio_managed specifies whether a filesystem is managed by GRIOv2 (true) or not (false ) if GRIOv2 has already been enabled on the node. The default is false. Setting grio_managed to false disables GRIO management for the specified filesystem, but it does not reset the grio_qual_bandwidth value. In this case, grio_qual_bandwidth is left unmodified in the cluster database and ignored.
grio_qual_bandwidth specifies a filesystem's qualified bandwidth in bytes (B suffix), kilobytes (KB), megabytes (MB), or gigabytes (GB), where the units are multiples of 1024. The default is MB for 4000 or less, B for 4001 or greater. If the filesystem is GRIO-managed, you must specify a qualified bandwidth with this attribute. To modify the qualified bandwidth, the filesystem must be unmounted.
For example, the following commands all create the myfs filesystem with a GRIOv2 qualified bandwidth of 1.2 GB/s (assuming that grio_managed=true has already been set):
cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1288500000 cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1258300KB cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1288.8MB cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1.2GB
Advanced-mode usage notes:
device is the name of the XVM volume (minus /dev/cxvm). The default is the filesystem name specified by name.
For example, to create a device name of mydev for the myfs filesystem:
cxfs_admin:mycluster> create filesystem name=myfs device=mydev
servers specifies the potential metadata servers that can serve the filesystem to the cluster. To specify multiple server-capable administration nodes, use a comma-separated list of node names.
The default is all server-capable administration nodes in the cluster. The list of potential metadata servers for a given filesystem is ordered, but because of network latencies and other unpredictable delays, it is impossible to predict which node will become the active metadata server for the initial mount of a filesystem.
Note: Before deleting a server-capable administration node, you must first unmount any filesystems for which the node is a potential metadata server. For example, to specify that either node2 or node3 could be the metadata server, with node2 being the primary server, for the myfs filesystem:
cxfs_admin:mycluster> create filesystem name=myfs servers=node2,node3
nodes specifies nodes that can mount the filesystem. If you do not specify the nodes attribute on the create command, all nodes can mount the filesystem.
You can later change the list of nodes allowed to mount the filesystem by using the modify command. For example, to restrict mounting the myfs filesystem to just nodes node1 and node2:
cxfs_admin:mycluster> create myfs nodes=node1,node2
To then add node3 (thereby allowing node1, node2, and node3 to mount the myfs filesystem):
cxfs_admin:mycluster> modify myfs nodes=+node3
To then prevent node1 from mounting (thereby allowing just node2 and node3 to mount the myfs filesystem):
cxfs_admin:mycluster> modify myfs nodes=-node1
mount_new_nodes specifies whether a newly created node will automatically mount the filesystem when it gets membership (true) or will not mount the filesystem (false). By default, new nodes mount all defined filesystems.
For example, to create filesystem myfs that is not automatically mounted by new nodes, use the following command:
cxfs_admin:mycluster> create filesystem name=myfs mount_new_nodes=false
To later mount the filesystem on node3 after it has been created, use the following command:
cxfs_admin:mycluster> mount myfs nodes=node3
For example, using prompting in basic mode:
cxfs_admin:clusterOne> create filesystem Specify the attributes for create filesystem: name? zj01s0 options? forced_unmount? false mountpoint? /mnt/zj01s0 mounted? true grio_managed? false Event at [ Dec 21 15:04:15 ] Filesystem "zj01s0" has been created, waiting for it to be mounted on all assigned nodes... Waiting for filesystem zj01s0, current status: A server is trying to mount Waiting for filesystem zj01s0, current status: cxfsxe5 trying to mount, penguin17 trying to mount, pg-27 trying to mount Waiting for filesystem zj01s0, current status: pg-27 trying to mount Operation completed successfully |
For example, using prompting in advanced mode:
cxfs_admin:clusterOne> create filesystem Specify the attributes for create filesystem: name? zj01s0 options? forced_unmount? false mountpoint? /mnt/zj01s0 device? zj01s0 servers? bert,cxfsxe5 nodes? bert,cxfsxe10,cxfsxe5,penguin17,pg-27 mounted? true mount_new_nodes? true grio_managed? false Event at [ Dec 21 15:02:28 ] Filesystem "zj01s0" has been created, waiting for it to be mounted on all assigned nodes... Waiting for filesystem zj01s0, current status: A server is trying to mount Waiting for filesystem zj01s0, current status: penguin17 trying to mount, pg-27 trying to mount Operation completed successfully |
| Note: After a filesystem has been defined in CXFS, running mkfs on it will cause errors to appear in the system log file. To avoid these errors, run mkfs before defining the filesystem in CXFS or delete the CXFS filesystem before running mkfs. See “Delete a CXFS Filesystem with cxfs_admin”. |
The mount command operates on the set of nodes that were specified in the nodes=[+/-]nodelist attribute when the filesystem was created. By default, this is all nodes in the cluster.
To mount the filesystem on all enabled nodes in the cluster:
mount filesystem |
To mount the filesystem on specific enabled nodes:
mount filesystem nodes=nodelist |
For example, to mount the filesystem myfs on only nodes node2 and node3:
cxfs_admin:mycluster> mount myfs nodes=node2,node3 |
To then add node4 (thereby allowing node2, node3, and node4 to mount the myfs filesystem):
modify myfs nodes=+node3 |
To then prevent node4 from mounting (thereby allowing just node2 and node3 to mount the myfs filesystem):
modify myfs nodes=-node4 |
| Note: When a node is configured to mount a filesystem, the node will automatically mount that filesystem when the node joins the cluster membership. |
To unmount a filesystem from all nodes in the cluster:
unmount filesystem |
| Note: As a convenience, cxfs_admin also accepts this command spelled as umount. |
To unmount the filesystem from a specific comma-separated list of nodes:
unmount filesystem nodes=[+/-]nodelist |
For example, to unmount filesystem myfs from nodes node1 and node3:
cxfs_admin:mycluster> unmount myfs nodes=node1,node3 |
| Note: When a node is configured to unmount a filesystem, the node will automatically unmount that filesystem when the node joins the cluster membership. |
The relocate command forcefully moves a filesystem's metadata server to another node in the cluster that has already been defined as a potential metadata server for that filesystem. This action is typically used to free a server so that it can be brought down for maintenance or upgrades. Relocation must also be explicitly enabled in the kernel with the cxfs_relocation_ok system tunable parameter (see “Relocation” in Chapter 1).
If relocation is explicitly enabled in the kernel, you can relocate a metadata server to another node by using the following command:
relocate filesystem server=new_metadata_server |
For example:
cxfs_admin:mycluster> relocate myfs server=node2 |
CXFS kernel membership is not affected by relocation. However, users may experience a degradation in filesystem performance while the metadata server is relocating.
Use the following command to delete a filesystem:
delete [filesystem:]filesystem |
You cannot delete a mounted filesystem; you must first unmount it. For example, if myfs is a unique name in the cluster database:
cxfs_admin:clusterOne> unmount zj01s0 Event at [ Dec 21 15:09:13 ] Waiting for filesystem "zj01s0" to be unmounted on all nodes Waiting for filesystem zj01s0, current status: bert trying to unmount, cxfsxe5 trying to unmount, penguin17 trying to unmount, pg-27 trying to unmount Waiting for filesystem zj01s0, current status: bert trying to unmount, penguin17 trying to unmount, pg-27 trying to unmount Waiting for filesystem zj01s0, current status: Unmounted Operation completed successfully cxfs_admin:clusterOne> delete zj01s0 Event at [ Dec 21 15:09:31 ] |
To show information about all filesystems:
show filesystem |
To show information about a specific filesystem:
show [filesystem:]fsname |
For example, for a filesystem named zj0ds0:
cxfs_admin:clusterOne> show zj01s0
Event at [ Dec 21 15:11:19 ]
filesystem:zj01s0:
device=zj01s0
forced_unmount=false
grio_managed=false
grio_qual_bandwidth=0B
mount=true
mount_new_nodes=true
mountpoint=/mnt/zj01s0
nodes:
bert, cxfsxe10, cxfsxe5, penguin17, pg-27
options=
servers:
bert, cxfsxe5
status:
blocksize=4.00KB
free=8.28GB
nodes:
bert=mounted
cxfsxe10=disabled
cxfsxe5=mounted
penguin17=mounted
pg-27=mounted
server=bert
size=8.37GB
stable=true
summary=Mounted
utilization=1% |
The filesystem status for a node can have one the following conditions:
disabled: the filesystem is configured to be mounted, but the node is disabled and therefore the filesystem cannot mount
inactive: the filesystem is configured to be mounted, but although the node is enabled it is not currently in membership (such as would be the case if CXFS services were stopped) and therefore the filesystem currently cannot mount
mounted: the filesystem is configured to be mounted and is currently mounted
trying to mount: the filesystem is configured to be mounted and is attempting to mount, but it may be having problems
unmounted: the filesystem is not configured to be mounted and is therefore not mounted
To limit the networks chosen from the private_net list or to reprioritize the list without making changes to the definition of each node, you can use the failover_net attribute.
| Caution: Do not configure failover_net while CXFS services are active; doing so can lead to cluster malfunction. |
Command syntax:
create failover_net network=IPaddress mask=netmask |
For example, suppose that you already have the following definitions for private_net for a three-node cluster:
nodeA:
10.0.0.1 10.1.0.1 192.168.0.1
nodeB:
10.0.0.2 10.1.0.2 192.168.0.2
nodeC:
10.0.0.3 10.1.0.3 192.168.0.3
If you want to limit the networks chosen to those using the 192.168.0.x subnet as the first priority and the 10.0.0.x subnet as the second priority, and not use the 10.1.0. x network at all, you would use the following commands:
cxfs_admin:mycluster> create failover_net network=192.168.0.0 mask=255.255.255.0 cxfs_admin:mycluster> create failover_net network=10.0.0.0 mask=255.255.255.0 |
| Note: For IPv6, you must supply the netmask
as the prefix number of bits. For example:
For IPv4, you can use either an IP address or the number of bits for the netmask. |
If the network specified for failover_net does not match any values specified for private_net, or if there are multiple interfaces specified for one node on a given network, cxfs-config will report an error.
This section discusses the following:
For general information, see “I/O Fencing” in Chapter 2.
| Note: Nodes without system controllers require I/O fencing to protect data integrity. A switch is mandatory to support I/O fencing; therefore, multiOS CXFS clusters require a switch. See the release notes for supported switches. |
To raise or lower a fence, see “Fencing Tasks with cxfs_admin”. For all of the options to update port information for switches, use the hafence command.
To define a new switch, use the following command:
create switch
name=switch_hostname
user=username (admin)
password=username_password (password)
vendor=brocade|qlogic|voltaire|lsi (brocade)
Advanced-mode:
[mask=ports_that_will_not_be_fenced]
[protocol=ssh|telnet] (ssh) |
| Note: You must define all of the switches within your fabric to which a CXFS client or server is connected. |
You will use a similar set of subcommands to modify a switch via the modify command.
Basic-mode usage notes:
name specifies the hostname of the Fibre Channel, SAS, or InfiniBand switch; this is used to determine the IP address of the switch.
password specifies the password for the specified username. The default is password.
Note: For passwordless authentication, the following public and private keys should be stored on each server-capable administration node: /root/.ssh/id_rsa.pub /root/.ssh/id_rsa user specifies the user name to use when sending a message to the switch. The default is admin.
vendor specifies the vendor of the switch. It can be one of the following values:
brocade (default) lsi qlogic voltaire For example, if myswitch is a QLogic switch:
cxfs_admin:mycluster> create switch name=myswitch vendor=qlogic
Advanced-mode usage notes:
mask specifies the ports on the switch that will never be fenced. By default, no ports are masked (and therefore all ports are available for fencing). The value for mask is a series of comma-separated port ranges. For example, the following states that ports 0, 4, and 12 through 15 for myswitch will never be fenced by CXFS:
cxfs_admin:mycluster> create switch name=myswitch mask=0,4,12-15
Note: For non-bladed switches, the port column of the switchShow output is the port number for CXFS.
For bladed switches, where the port number is not unique, the port column of switchShow output means the port number of one slot. However, CXFS requires a unique number identified by the combination of slot and port for bladed switch, which is usually represented by the Index or Area value in the switchShow output (depending upon the switch version). For example, the switchShow output below for the Brocade 48000 switch indicates that you would use a mask value of 16 for port 0 in slot 2:brocade48000:admin> switchShow Index Slot Port Address Media Speed State Proto =================================================== 0 1 0 010000 id N4 Online F-Port 10:00:00:00:c9:5f:9b:ea 1 1 1 010100 id N4 Online F-Port 10:00:00:00:c9:5f:ab:d9 .... 142 1 30 018e00 id N4 Online F-Port 50:06:0e:80:04:5c:0b:46 143 1 31 018f00 id N4 Online F-Port 50:06:0e:80:04:5c:0b:66 16 2 0 011000 id N4 Online F-Port 10:00:00:00:c9:5f:a1:f5 17 2 1 011100 id N4 Online F-Port 10:00:00:00:c9:5f:a1:72 ...
For more details, see the switch documentation.Server-capable administration nodes automatically discover the available HBAs and, when fencing is triggered, fence off all of the SAN HBAs when the Fence or FenceReset fail policy is selected. However, masked HBAs will not be fenced. Masking lets you prevent the fencing of devices that are attached to the SAN but are not shared with the cluster, to ensure that they remain available regardless of CXFS status. You would want to mask HBAs used for access to tape storage, or HBAs that are only ever used to access local (nonclustered) devices.
protocol specifies the protocol to use when sending a message to the switch, either ssh or telnet. The ssh option is only valid with Brocade and InfiniBand switches. The default is telnet .
To delete a switch, use the following command:
delete [switch:]switch_hostname |
For example, if myswitch is a unique name in the cluster database:
cxfs_admin:mycluster> delete myswitch |
To display all of the switches in the system, use the following command:
show switch [output=full_pathname_of_output_file] |
For example, in basic mode (truncated):
cxfs_admin:clusterOne> show switch
Event at [ Dec 21 15:20:59 ]
switch:
brocade26cp1:
hostname=brocade26cp1
num_ports=192
port:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50,
...
vendor=brocade |
To send the output to the /tmp/switchinfo file:
cxfs_admin:mycluster> show switch output=/tmp/switchinfo |
To display a specific switch:
show [switch:]switchname [output=full_pathname_of_output_file] |
To display mask values, use advanced mode (see “Basic and Advanced Mode”.) For example, if myswitch is a unique name in the cluster database (truncated):
cxfs_admin:clusterOne> show switch
Event at [ Dec 04 09:34:31 ]
switch:
brocade26cp0:
hostname=brocade26cp0
num_ports=256
port:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, ...
249, 250, 251, 252, 253, 254, 255
protocol=ssh
vendor=brocade |
To display the switches and ports each host is connected to:
show wwns |
For example:
cxfs_admin:clusterOne> show wwns
Event at [ Dec 21 15:26:11 ]
node:bert:status:wwns:
100000062b126ff8:
order=1
100000062b126ff9:
order=0
switch=brocade26cp1
switch_port=62
node:cxfsxe5:status:wwns:
210000e08b123d95:
order=0
switch=brocade26cp1
switch_port=167
node:penguin17:status:wwns:
210000e08b1a07d8:
order=0
switch=brocade26cp1
switch_port=41
node:pg-27:status:wwns:
210000e08b1284c6:
order=0
switch=brocade26cp1
switch_port=170
210100e08b3284c6:
order=1 |
To show full status details for each port on the switch, use one of the following commands:
show [switch:]switchname all show switchname:port |
For example, for the switch named brocade26cp1:
cxfs_admin:clusterOne> show brocade26cp1:port
Event at [ Dec 21 15:27:01 ]
switch:brocade26cp1:port:
0:
status=Enabled
wwn=100000062b117c8c
1:
status=Enabled
wwn=100000062b117954
2:
status=Enabled
wwn=100000062b117c7c
3:
status=Enabled
wwn=100000062b11796c
... |
This section discusses the following fencing tasks:
To query the fencing status of all nodes, enter the query fence command when cxfs_admin is executed directly on a server-capable administration node. For example:
cxfs_admin:clusterOne> query fence
Event at [ Dec 21 15:29:07 ]
Waiting for shell command to end
Switch[0] "brocade26cp1" has 192 ports
Port 41 type=FABRIC status=enabled hba=210000e08b1a07d8 on host penguin17
Port 62 type=FABRIC status=enabled hba=100000062b126ff9 on host bert
Port 167 type=FABRIC status=enabled hba=210000e08b123d95 on host cxfsxe5
Port 170 type=FABRIC status=enabled hba=210000e08b1284c6 on host pg-27
Operation completed successfully |
To raise the I/O fence for a node, enter the following when cxfs_admin is executed directly on a server-capable administration node:
cxfs_admin:mycluster> raise fence node=nodename |
To be prompted for required attributes, you must be in advanced mode.
Raising an I/O fence isolates the node from the SAN; CXFS sends a messages via the telnet (default) or ssh protocol to the switch and disables the port. After the node is isolated, it cannot corrupt data in the shared CXFS filesystem.
For example, to isolate node pg-27:
cxfs_admin:clusterOne> raise fence node=pg-27 Event at [ Dec 21 15:29:47 ] Waiting for shell command to end Operation completed successfully |
To lower the I/O fence for a node, enter the following when cxfs_admin is executed directly on a server-capable administration node:
cxfs_admin:mycluster> lower fence node=nodename |
To be prompted for required attributes, you must be in advanced mode.
Lowering the fence reenables the port and allows the node to reconnect to the SAN and access the shared CXFS filesystem.
For example, to reconnect nodepg-27:
cxfs_admin:clusterOne> lower fence node=pg-27 Event at [ Dec 21 15:30:37 ] Waiting for shell command to end Operation completed successfully |
This section discusses the following:
The cxfs_admin config command displays a series of commands that represent the current configuration of the objects specified. You can use this output to recreate the configuration of the entire cluster or a subset of it.
By default, config displays information at the cxfs_admin prompt. To write the configuration output to a file, use the output attribute and specify the full pathname or relative pathname of the file to contain the information:
config node output=pathname |
You can use the generated file with the -f command line option to recreate the configuration at a later time.
| Note: For a more-readable configuration output (without the related commands), use the show command rather than the config command. |
To create cxfs_admin scripts, do the following:
Dump the entire cluster configuration to a file, such as /tmp/config1:
cxfs_admin:mycluster> config * output=/tmp/config1
Edit the /tmp/config1 file:
Modify the output so that the first node created is the server-capable administration node from which you are going to execute the cxfs_admin command to create the cluster. (By default, the cxfs_admin config command output lists nodes in alphabetical order by node name without regard to node type.)
Cut the create filesystem command lines at the end of the file and paste them into a second script file, such as /tmp/config2.
To recreate the cluster using cxfs_admin scripts, do the following:
Verify that the cluster configuration is empty by using the cxfs-config command. For example:
server-admin# /usr/cluster/bin/cxfs-config Global: cluster: <not defined> (id <n/a>) tiebreaker: <n/a> dynamic heartbeat: <n/a> Networks: <none> Machines: <none> Autoconf: <none> Filesystems: <none> Switches: <none> cxfs-config warnings/errors: no cluster configured no machines defined
Initialize the cluster and recreate the base cluster (except for the CXFS filesystems) by using the first script file (/tmp/config1):
server-admin# /usr/cluster/bin/cxfs_admin -s -f /tmp/config1
Note: Do not reboot or restart CXFS or cluster services before this script finishes. Reboot the server-capable administration node:
server-admin# reboot
Repeat the reboot command on each server-capable administration node in the cluster.
Recreate the CXFS filesystems by using the second script file (/tmp/config2), providing the cluster name that was established as a result of step 2:
server-admin# /usr/cluster/bin/cxfs_admin -f /tmp/config2