This chapter discusses the following:
 | Note: The example output might not exactly match the output of your system. |
For an overview of the tasks that must be performed to configure a cluster, see “Initial Setup with the cxfs_admin Command” in Chapter 9.
You can also use the clconf_info tool to view status. See Chapter 14, “Monitoring Status”.
For help with error messages, see “cxfs_admin Errors” in Chapter 15.
See also Appendix G, “Migration from cmgr to cxfs_admin”.
To use the cxfs_admin command, you must be logged to a node that has permission to access the CXFS cluster database; to make configuration changes using cxfs_admin, you must be logged in as root. See “Setting cxfs_admin Access Permissions”.
| Note: For the steps to create a cluster for the first time, see “Initial Setup with the cxfs_admin Command” in Chapter 9. |
This section discusses the following:
To start cxfs_admin, enter the following:
/usr/cluster/bin/cxfs_admin |
See also:
Some cxfs_admin commands affect the cxfs_admin operating environment itself, some display status information, and others affect objects or classes. Within cxfs_admin, an object is a specific item that is configured in the CXFS cluster and a class contains a group of similar objects. For example, the filesystem names fs1 and fs2 would both be objects within the filesystem class.
Within a class, all objects must have unique names. If all objects in the cluster have unique names, you can abbreviate some commands by omiting the class name. However, if two or more objects in the cluster database have the same name, you must specify the class in order to uniquely identify the object.
The basic command syntax is:
command [[class:]object] [attributes] |
where attributes takes a number of forms depending on the context:
attribute attribute=value attribute=value1,value2,value3... |
The actual syntax components for any given command varies, based on the needs of the command. For example, the following command requires no parameters to see a summary of the cluster:
cxfs_admin:mycluster> show |
If an object name is unique within the cluster database, you can omit its class name. For example, if the name nodeA is unique within the database:
cxfs_admin:mycluster> show nodeA |
However, if there were multiple objects named production , you must enter the class name:
cxfs_admin:mycluster> show node:production |
Classes and objects may include the following shell-style wildcard characters:
* ? [...] |
| Note: You can use the asterisk (*) alone only
with the show and configcommands,
which do not make changes to the database. For these commands, you can
use a * character in place of the object
to apply the command to entire cluster (if you do not specify
any attributes, you can omit the * character.)
As a safety measure, however, using a wildcard in other instances requires that you also include at least one other character in the object's name; for example, if you wanted to unmount both myfileA and yourfileB, you could enter unmount *file* . |
With the show and config commands,
Command names and attribute names are not case-sensitive. However, all attribute values other the node name (in create node nodename) and the switch name (in create switch switchname) are case sensitive.
You can see possible attributes by pressing the <TAB> key after entering the command or object. For example:
cxfs_admin:mycluster> create filesystem <TAB>
Required attributes:
name= : A string
Optional attributes:
forced_unmount= : True/false or enabled/disabled (default is "false")
mounted= : True/false or enabled/disabled (default is "true")
mountpoint= : A pathname
options= : Nothing, one or more strings (can be empty) |
The required attributes are listed first followed by optional attributes. The list of attributes will differ depending upon whether you are in basic or advanced mode; see “Basic and Advanced Mode”.
Partially typing in the attribute name and pressing <TAB> will complete the attribute name if unique, or show a list of matching attribute names. To see what kind of values are required for an attribute, press <TAB> after the = sign. For example:
cxfs_admin:mycluster> create node os=<TAB> AIX IRIX Linux MacOSX Solaris Unknown Windows |
Use $ to refer to the object in the last command.
For example, to delete nodeA, if it has a unique name within the cluster database:
cxfs_admin:mycluster> disable nodeA cxfs_admin:mycluster> delete $ |
To specify multiple objects, separate them with a comma. For example:
cxfs_admin:mycluster> show nodeA,nodeB |
You can abbreviate commands, objects, and attributes by entering in the first character or two followed by pressing the <TAB> key. If more than one match is available, cxfs_admin shows a list of the possible matches.
At any time, you can enter help or ? to see help text.
To see help for a given topic:
help topicname |
The list of general topics includes the following:
attributes commandline commands cxfs objects overview setup syntax tasks waiting |
There is also help for each cxfs_admin command. For example, to see help about the create command:
cxfs_admin:mycluster> help create |
To see all of the available help topics, press the <TAB> key:
cxfs_admin:mycluster> help <TAB> |
To see a list of available commands for an object, such as a class like filesystem or a specific instance of a class like the filesystem myfs, use the ops command:
ops object |
For example:
cxfs_admin:mycluster> ops filesystem
Event at [ Jan 21 15:58:02 ]
Commands for "filesystem":
config, create, ops, show
cxfs_admin:mycluster> ops myfs
Event at [ Jan 21 15:58:03 ]
Commands for "filesystem:myfs":
config, delete, modify, mount, ops, relocate, show, unmount |
The cxfs_admin tool only allows one user to use cxfs_admin to make changes to the cluster database at a time. If you are the first person to invoke cxfs_admin, you automatically get the lock. If someone else already has the lock, you will enter in read-only mode. If you are in read-only mode, it is reflected in the cxfs_admin prompt.
To forcefully obtain the lock from someone else, you can use the steal attribute with the lock command. For example:
cxfs_admin:mycluster (read only) > lock Event at [ Jan 21 15:58:02 ] The administration lock is already held by root@node2 (pid=48449) cxfs_admin:mycluster (read only) > lock steal=true Event at [ Jan 21 15:58:03 ] The administration lock has been stolen from root@node2 (pid=48449) cxfs_admin:mycluster> |
If someone holds the lock while you are using cxfs_admin but later drops it, there is no need to steal the lock.
If you want to manually enter read-only mode, use the unlock command. For example:
cxfs_admin:mycluster> unlock Event at [ Jan 21 15:58:02 ] cxfs_admin:mycluster (read only) > |
 | Caution: The cxfs_admin lock does not prevent other users from using the CXFS GUI while cxfs_admin is running. You should make database changes with only one instance of the CXFS GUI or locked cxfs_admin commands at any given time. |
The cxfs_admin operates in two modes:
Basic, which only shows the common options and attributes in show output, provides a list of possible choices when using the <TAB> key, and prompting mode.
Advanced, which provides a list of possible choices when using the <TAB> key, prompts for all possible fields, displays all attributes, and includes debugging information in output. Advanced-mode commands and attributes are not included in prompts or <TAB> key completion when you are in basic mode. However, you can still manually enter an advanced attribute if you know it, even in basic mode.The advanced commands and attributes are noted in their help topics.
Note: You should only use the advanced-mode commands and attributes at the advice of SGI support. Using the advanced mode commands or changing advanced mode attributes may induce unexpected behavior.
You can enter advanced mode by using cxfs_admin -a on the command line or by entering the following cxfs_admin command:
cxfs_admin:mycluster> set mode=advanced |
To return to basic mode:
cxfs_admin:mycluster> set mode=basic |
For example, the following output shows only the basic-mode information for the node named cxfsxe5:
cxfs_admin:clusterOne > set mode=basic
Event at [ Jan 21 15:35:23 ]
Mode is set to basic
cxfs_admin:clusterOne> show cxfsxe5
Event at [ Jan 21 15:35:24 ]
node:cxfsxe5:
autoconf=false
cellid=0
clustername=clusterOne
enabled=true
failpolicy=Fence,Shutdown
hostname=cxfsxe5.mycompany.com
nodeid=1
os=Linux
private_net:
10.0.199.1
status:
age=17
build=13:29:27 Jan 14 2009
connected=true
fencing=Stable
license:
have_license=true
member=true
stable=true
summary=Stable
version=5.4.0.1
wwns:
210000e08b123d95
type=server_admin |
| Note: The have_license field reports licenses for client-only nodes that have been obtained from the server. It does not display information about the server licenses. For more information about licensing, see “Show License Information with cxfs_admin”. |
For example, the following output shows all information that is available in advanced mode for cxfsxe5:
cxfs_admin:clusterOne > set mode=advanced
Event at [ Jan 21 15:35:25 ]
Mode is set to advanced
cxfs_admin:clusterOne> show cxfsxe5
Event at [ Jan 21 15:35:26 ]
node:cxfsxe5:
autoconf=false
cellid=0
clustername=clusterOne
enabled=true
failpolicy=Fence,Shutdown
hostname=cxfsxe5.mycompany.com
nodeid=1
os=Linux
private_net:
10.0.199.1
status:
age=17
build=13:29:27 Jan 14 2009
connected=true
fencing=Stable
license:
have_license=true
member=true
stable=true
summary=Stable
version=5.4.0.1
wwns:
210000e08b123d95
type=server_admin |
| Note: If a client is not connected to the cluster, the build and version fields will not display because the node cannot respond to for requests for this information. |
Some cxfs_admin commands will prompt you for required attributes if you press ENTER after the command name. To see information about the legal values for an attribute, press <TAB> after a question.
For example:
cxfs_admin:mycluster> create
What do you want to create? The following can be used:
cluster, failover_net, filesystem, node, switch
create what? node
Specify the attributes for create node:
name? mynode
type? client_only
os?
AIX IRIX Linux MacOSX Solaris Unknown Windows
os?
... |
In basic mode, you are only prompted for required parameters. To be prompted for all possible parameters, use advanced mode. See “Basic and Advanced Mode”.
Depending upon the context, cxfs_admin prompts will vary based upon your answers to previous prompts. For example, if you specify that a node's os value is MacOSX , cxfs_admin will not prompt you for the type because Mac OS X nodes are required to be client-only nodes.
To exit from prompt mode, sent an interrupt signal (typically, press Ctrl-C).
The history command displays a list of commands that have been used in cxfs_admin since it was started:
Display all of the commands (up to the previous 1000 commands):
history
Limit the commands to the last specified number of items:
history num=number_of_items
For example, to display only the last 10 commands:
cxfs_admin:mycluster> history num=10
Clear the history:
history clear
Send the history to a file (you must enter the full pathname of the file):
history output=full_pathname
For example, to send the history output to the file /tmp/myhistory :
cxfs_admin:mycluster> history output=/tmp/myhistory
Some commands in cxfs_admin take a noticeable period of time to complete. cxfs_admin displays informational updates as a command progresses or a period character if nothing has changed within 2 seconds.
After 1 minute without change, a command will terminate. This may happen when there is a problem in creating or modifying a node or filesystem. The update message shows the problem status.
To interrupt a command, send an interrupt signal (usually Ctrl-C).
You can enter cxfs_admin commands directly from the cxfs_admin command line by using the following format:
server-admin# /usr/cluster/bin/cxfs_admin -c "cxfs_admin_commands" |
For example, to display information about the cluster (line breaks shown for readability):
cxfs_admin:cc_test2> show cluster
Event at [ Jan 21 15:58:02 ]
cxfs:cluster:
cc_test2:
access:
admin=server
monitor=
autoconf:
allowed:
(none)
disallowed:
(none)
failover_net:
10.11.0.0, 128.162.242.0
filesystem:
concatfs, mirrorfs, stripefs
node:
cc-xe, cc-xe2, aiden, brenna, cc-mac1, cc-vista, cc-win2003,
cc-win64, cc-winxp, cxfsibm2, cxfssun4, gaeth, liam,
lorcan, minnesota, nasca, padraig
status:
filesystems:
summary=concatfs: cxfsibm2 trying to mount
stripefs: cxfsibm2 trying to mount
licenses:
cxfs_client
nodes:
summary=cc-vista: Inactive
cc-win64: Inactive
cxfsibm2: Mounted 0 of 2 filesystems
summary=node(s) not stable, filesystem(s) not stable
switch:
fcswitch12, fcswitch13
tiebreaker=minnesota |
You can execute a series of cxfs_admin commands by using the -f option and specifying an input file:
server-admin# /usr/cluster/bin/cxfs_admin -f command_file |
For example, suppose the file /tmp/showme contains the following:
cxfs6# more /tmp/showme show cluster show filesystem |
You can execute the following command, which will yield the indicated output: (line breaks shown for readability):
# /usr/cluster/bin/cxfs_admin -f /tmp/showme
Event at [ Jan 21 15:58:02 ]
Connecting to the local CXFS server...
cxfs:cluster:
cc_test2:
access:
admin=server
monitor=
failover_net:
10.11.0.0, 128.162.242.0
filesystem:
concatfs, mirrorfs, stripefs
node:
cc-xe, cc-xe2, aiden, brenna, cc-mac1, cc-vista, cc-win2003, cc-win64,
cc-winxp, cxfsibm2, cxfssun4, gaeth, liam, lorcan, minnesota,
nasca, padraig
status:
filesystems:
summary=concatfs: cxfsibm2 trying to mount
stripefs: cxfsibm2 trying to mount
licenses:
cxfs_client
nodes:
summary=cc-vista: Inactive
cc-win64: Inactive
cxfsibm2: Mounted 0 of 2 filesystems
summary=node(s) not stable, filesystem(s) not stable
switch:
fcswitch12, fcswitch13
tiebreaker=minnesota
filesystem:
concatfs:
forced_unmount=true
mount=true
mountpoint=/mnt/concatfs
nodes:
aiden, brenna, cc-mac1, cc-vista, cc-win2003, cc-win64, cc-winxp,
cc-xe, cc-xe2, cxfsibm2, cxfssun4, gaeth, liam, lorcan, minnesota,
nasca, padraig
options=rw
servers:
cc-xe, cc-xe2
status:
free=918GB
nodes:
aiden=mounted
brenna=mounted
cc-mac1=mounted
cc-vista=inactive
cc-win2003=mounted
cc-win64=inactive
cc-winxp=mounted
cc-xe=mounted
cc-xe2=mounted
cxfsibm2=trying to mount
cxfssun4=mounted
gaeth=mounted
liam=mounted
lorcan=mounted
minnesota=mounted
nasca=mounted
padraig=mounted
server=cc-xe2
size=930GB
summary=cxfsibm2 trying to mount
utilization=1%
mirrorfs:
forced_unmount=true
mount=false
mountpoint=/mnt/mirrorfs
nodes:
aiden, brenna, cc-mac1, cc-vista, cc-win2003, cc-win64, cc-winxp,
cc-xe, cc-xe2, cxfsibm2, cxfssun4, gaeth, liam, lorcan, minnesota,
nasca, padraig
options=rw
servers:
cc-xe, cc-xe2
status:
nodes:
aiden=unmounted
brenna=unmounted
cc-mac1=unmounted
cc-vista=inactive
cc-win2003=unmounted
cc-win64=inactive
cc-winxp=unmounted
cc-xe=unmounted
cc-xe2=unmounted
cxfsibm2=unmounted
cxfssun4=unmounted
gaeth=unmounted
liam=unmounted
lorcan=unmounted
minnesota=unmounted
nasca=unmounted
padraig=unmounted
summary=Unmounted
stripefs:
forced_unmount=false
mount=true
mountpoint=/mnt/stripefs
nodes:
aiden, brenna, cc-mac1, cc-vista, cc-win2003, cc-win64, cc-winxp,
cc-xe, cc-xe2, cxfsibm2, cxfssun4, gaeth, liam, lorcan, minnesota,
nasca, padraig
options=rw,dmi
servers:
cc-xe, cc-xe2
status:
free=3.37TB
nodes:
aiden=mounted
brenna=mounted
cc-mac1=mounted
cc-vista=inactive
cc-win2003=mounted
cc-win64=inactive
cc-winxp=mounted
cc-xe=mounted
cc-xe2=mounted
cxfsibm2=trying to mount
cxfssun4=mounted
gaeth=mounted
liam=mounted
lorcan=mounted
minnesota=mounted
nasca=mounted
padraig=mounted
server=cc-xe2
size=3.63TB
summary=cxfsibm2 trying to mount
utilization=7% |
You can use one of the following methods to set the defaults for the way cxfs_admin behaves and the editor to use within cxfs_admin , in the following order of precedence:
Use the set command within cxfs_admin :
set [editor=emacs|vi] (emacs) [line_wrap=true|false] (true) [mode=basic|advanced] (basic) [stop_on_error=true|false] (true)
For example, to change to vi:
cxfs_admin:mycluster> set editor=vi
Usage notes:
editor specifies the editor style ( emacs or vi). The default is emacs .
line_wrap specifies the ability to wrap a line at the edge of the current window (true) or no line wrap (false). The default is true .
mode determines whether all values (advanced) or only those values that are required (basic). The default is basic. See “Basic and Advanced Mode”.
stop_on_error will abort a command upon encountering an error (true) or keep going ( false). The default is true.
Set the following environment variables:
Environment Variable Values
CXFS_ADMIN_CLUSTER_NAME clustername. Setting this value lets you bypass using the -i option if you have multiple clusters connected to the same public network. There is no default.
CXFS_ADMIN_EDITOR emacs (default) or vi
CXFS_ADMIN_LINE_WRAP true (default) or false
CXFS_ADMIN_MODE basic (default) or advanced
CXFS_ADMIN_STOP_ON_ERROR true (default) or false
Use the .cxfs_admin file in your home directory (as defined by the $HOME environment variable) to set the following:
mode=basic|advanced cluster_name=clustername editor=emacs|vi stop_on_error=true|false line_wrap=true|false
Lines within the .cxfs_admin file that begin with the # character or a space are ignored, as are lines that do not contain the = character.
For example, to use the mycluster cluster in advanced mode and the vi editor:
# My settings for cxfs_admin: cluster=mycluster mode=advanced editor=vi
The access command allows you to specify hosts that have permission to modify the cluster configuration and hosts that have permission to monitor the cluster state:
access
allow=hostname_or_IPaddress_list
permission=admin|monitor (monitor, only available with allow)
deny=server_name |
By default, all server-capable administration nodes in the cluster are granted admin access (without using the access command).
For example, to grant remotehostA and remotehostB permission to modify the cluster configuration:
cxfs_admin:mycluster> access allow=remotehostA,remotehostB permission=admin |
To grant read-only rights in order to monitor to the cluster configuration and status (monitor is the default access level):
cxfs_admin:mycluster> access allow=remotehostA |
To revoke all access to the cluster database for a host that was previously granted some level of access, use the following command:
cxfs_admin:mycluster> access deny=remotehostA,remotehostB |
To view the current access rights, use the following command:
show access |
For example:
cxfs_admin:mycluster> show access
Event at [ Jan 21 15:58:02 ]
access:
admin=server
monitor=cluster |
Usage notes:
allow specifies the hosts to be granted the specified permission. These hosts must be on the same private network as the cluster nodes. To specify multiple hosts, use a comma-separated list. There are three reserved hostnames:
cluster denotes any node defined in the cluster
server denotes any server-capable administration node, even one that is disabled from CXFS membership (see “Disable a Node with cxfs_admin”)
any denotes any system that is on the private network
permission specifies read/write access (admin) or read-only access (monitor). The default is monitor.
deny specifies the hosts to be denied all access to the cluster database (for hosts that were previously granted some level of access). To specify multiple hosts, use a comma-separated list. The same reserved hostnames as allow apply.
If you have multiple clusters connected to the same public network, use the -i option to identify the cluster name:
cxfs_admin -i clustername |
For example, for the cluster named mycluster:
server-admin# /usr/cluster/bin/cxfs_admin -i mycluster |
On each client-only node, add -i clustername to the cxfs_client.options file. For more information about the cxfs_client.options file, see CXFS 5 Client-Only Guide for SGI InfiniteStorage.
| Note: CXFS does not support multiple clusters on the same private network. |
This section discusses the following:
| Note: The entire cluster status information is sent to each server-capable administration node each time a change is made to the cluster database; therefore, the more server-capable administration nodes in a configuration, the longer it will take. |
The autoconf class creates a rule that determines whether the specified new client-only nodes can be automatically configured into the CXFS cluster database. This is known as easy client configuration.
| Note: The autoconf action only applies to hosts that are not currently defined in the cluster database. |
You can specify a single hostname and/or a range of IP addresses to be allowed (whitelisted) or disallowed ( blacklisted). Only those nodes listed in a whitelist and not listed in a blacklist can be automatically configured. (That is, the blacklist has higher priority than the whitelist.)
Each autoconf rule must have a unique name. By default, only nodes that are defined by an autoconf rule set to enable_node=true will be defined as enabled and will be allowed to join the cluster. (If you set enable_node=false , the node will be defined in the database but disabled/inactive.)
To create a new rule:
create autoconf rule_name=rulename policy=allowed|disallowed [hostname=hostname] [startIP=startingIPaddress endIP=endingIPaddress] [enable_node=true|false]
To change an existing rule:
modify [allowed:|disallowed:]rulename
To delete an existing rule:
delete [allowed:|disallowed:]rulename
To show an existing rule:
show [allowed:|disallowed:]rulename
To show all existing rules:
show autoconf
To show all allowed rules, which displays the rules for client-only nodes that are allowed to be automatically configured (whitelisted):
show allowed
To show all disallowed rules, which displays the rules for client-only nodes that are explicitly not allowed to be automatically configured (blacklisted):
show disallowed
For example, suppose you want to let nodes NodeA and NodeB to be automatically configured and enabled in the cluster. You would create two rules:
cxfs_admin:mycluster> create autoconf rule_name=ruleNodeA policy=allowed hostname=NodeA enable_node=true cxfs_admin:mycluster> create autoconf rule_name=ruleNodeB policy=allowed hostname=NodeB enable_node=true |
If you then wanted to change the rule so that NodeA could not be automatically configured, you would modify the autoconf rule named ruleNodeA:
cxfs_admin:mycluster> modify autoconf rule_name=ruleNodeA policy=disallowed hostname=NodeA |
| Note: If NodeA was already defined in the cluster database, this would have no affect. The autoconf rules apply only to nodes that are not currently defined. |
In this case, you could also delete ruleNodeA, because not listing node in an autoconf rule is the same as disallowing it.
Suppose all of your client-only nodes have IP addresses within the range 192.168.0.1 through 192.168.0.20. To let all of the nodes be configured automatically except for the node named bert, you would need to create two rules:
cxfs_admin:mycluster> create autoconf rule_name=allowall policy=allowed startIP=192.168.0.1 endIP=192.168.0.20 enable_node=true cxfs_admin:mycluster> create autoconf rule_name=rulebert policy=disallowed |
If there is a node that is not within the IP address range that you also want to allow, you can include it in the same rule. For example, the following would allow all nodes in the range 192.168.0.1 through 192.168.0.20 plus the node named ernie (that is in a different IP range):
cxfs_admin:mycluster> create autoconf rule_name=allowall policy=allowed startIP=192.168.0.1 endIP=192.168.0.20 hostname=ernie enable_node=true |
After you have finished creating or modifying all of the desired autoconf rules, you must unlock all cxfs_admin sessions in order for nodes to be automatically configured. (The automatic configuration process must have access to the cxfs_admin lock.) If the node that is created or reconfigured via autoconf is unable to join cluster membership, you must reboot the node.
| Note: Before adding a server-capable administration node, you must first unmount any filesystems for which the node will be a potential metadata server. |
To define a node, use the following command and attributes (line breaks shown here for readability, defaults in parentheses):
create node name=nodename type=client_only|server_admin (client_only) os=AIX|IRIX|Linux|MacOSX|Solaris|Windows|Unknown (Linux) private_net private_network_IPaddress_list|hostname_list Advanced-mode: enabled=true|false (true) failpolicy=FenceReset,Fence,Reset,Shutdown (Fence,Shutdown) hostname=logical_hostname (fully_qualified_domain_name_of_nodename) nodeid=nodeID (assigned by cxfs_admin) partition_id=partition_number reset_method=nmi|powerCycle|reset (powerCycle) reset_port=l1|l2|bmc|msc|mmsc reset_password=password reset_status=enabled|disabled (enabled) reset_node=node_sending_reset_command reset_comms=tty|network|ipmi reset_device=port|IP_address_or_hostname_of_device |
When you create a client-only node, it will by default automatically be enabled and join the cluster. When adding the first server-capable administration node, you must restart it or restart CXFS services and cluster services on the node:
server-admin# service /cxfs stop server-admin# service /cxfs_cluster stop server-admin# service /cxfs_cluster start server-admin# service /cxfs start |
To use prompting mode, press <ENTER>. To obtain information about legal values, press <TAB>.
For example, to create a client-only node, you could do the following, pressing the <TAB> key to see the list of operating system values:
cxfs_admin:mycluster> create node Specify the attributes for create node: name? newnode type? client_only os? <TAB> AIX IRIX Linux MacOSX Solaris Unknown Windows os? irix private_net? 192.168.0.178 Event at [ Jan 21 15:58:02 ] Node "newnode" has been created, waiting for it to join the cluster... Waiting for node newnode, current status: Inactive Waiting for node newnode, current status: Establishing membership Waiting for node newnode, current status: Probing XVM volumes Operation completed successfully |
| Note: When you add a Linux node, cxfs_admin must prompt you for the type value because that node could be of type server-capable or client-only. |
To create a server-capable administration node using the defaults, you must delete the client_only default for type and enter server_admin. For example:
cxfs_admin:mycluster> create node Specify the attributes for create node: name? newnode type? server_admin private_net? 192.168.0.178 Event at [ Jan 21 15:58:02 ] Node "newnode" has been created, waiting for it to join the cluster... Please restart all cxfs and cluster services on the server "newnode" to make it join the cluster. |
To create a server-capable administration node in advanced mode, which can prompt you to set additional values, such as for reset_method and failpolicy:
cxfs_admin:mycluster> set mode=advanced cxfs_admin:mycluster> create node Specify the attributes for create node: name? newnode type? server_admin private_net? 192.168.0.178 enabled? true failpolicy? Reset,Shutdown hostname? newnode.example.com nodeid? 1 partition_id? reset_method? reset reset_port? l2 reset_password? reset_status? enabled reset_node? node2 reset_comms? network reset_device? newnode-l2.mycompany.com Event at [ Jan 21 15:58:02 ] Node "newnode" has been created, waiting for it to join the cluster... Please restart all cxfs and cluster services on the server "newnode" to make it join the cluster. |
You will use a similar set of subcommands to modify a node. (To modify a node, you must first disable it.)
To modify the failpolicy to eliminate Shutdown:
cxfs_admin:mycluster> modify newnode failpolicy=Reset,Fence |
Basic-mode usage notes:
name is a simple hostname (such as lilly) or a fully qualified domain name (such as lilly.example.com ) or an entirely different name (such as node1). It cannot begin with a number or an underscore (_), or include any whitespace characters, and can be at most 255 characters.
type specifies the function of the node. Enter one of the following:
client_only is a node that shares CXFS filesystems but will never be a CXFS metadata server. Most nodes should be client-only nodes. AIX, MacOSX, Solaris , Windows and Unknown nodes are automatically specified as client-only and you will not be prompted for this value for these operating systems.
server_admin is an SGI Foundation Software (type linux) node that is a potential CXFS metadata server. (You will use the create filesystem command to define the specific filesystem for which this node can be a metadata server.)
os is one of the following for client-only nodes:
AIX IRIX Linux (SGI Foundation Software or Linux third-party) MacOSX Solaris Unknown Windows private_net is the IP address or hostname of the private network. (The hostname must be resolved in the /etc/hosts file.) SGI requires that this network be private; see “Private Network” in Chapter 1.
There can be up to 8 network interfaces. The order in which you specify the interfaces determines their priority; the first interface will be priority 1, the second priority 2, and so on. There is no default.
To inform the servers of the failover networks, you must create a failover_net network. See “Network Failover Tasks with cxfs_admin”.
For more information about using the hostname, see “Hostname Resolution and Network Configuration Rules” in Chapter 6.
Advanced-mode usage notes:
enabled determines if a node will be able to obtain CXFS membership (true) or not ( false). By default, the new node is enabled (true). To enable a command created with enabled=false, use the enable command. See “Enable a Node with cxfs_admin”.
failpolicy determines what happens to a failed node. You can specify up to three methods. The second method will be completed only if the first method fails; the third method will be completed only if both the first and second options fail. Separate options by commas (not whitespace). The option choices are as follows:
Fence disables access to the SAN from the problem node. Fencing provides faster recovery of the CXFS kernel membership than reset.
FenceReset performs a fence and then, if the node is successfully fenced, also performs an asynchronous reset of the node via a system controller; recovery begins without waiting for reset acknowledgement.
Note: SGI recommends that a server-capable administration node include Reset in its failpolicy (unless it is the only server-capable administration node in the cluster). See “Data Integrity Protection” in Chapter 1. The FenceReset and Fence policies are mutually exclusive.
Reset performs a system reset via a system controller. This action requires a reset_method value; see “Hardware and Software Requirements for Server-Capable Administration Nodes” in Chapter 1.
Shutdown tells the other nodes in the cluster to wait for a period of time (long enough for the node to shut itself down) before reforming the CXFS kernel membership. (However, there is no notification that the node's shutdown has actually taken place.)
Caution: Because there is no notification that a shutdown has occurred, if you have a cluster with no tiebreaker, you must not use the shutdown setting for any server-capable administration node in order to avoid split clusters being formed. See “Shutdown” in Chapter 2. You should not use the Shutdown fail policy on client nodes if you choose dynamic CXFS kernel heartbeat monitoring for the cluster.
Note: If the failure hierarchy contains reset or fencereset, the reset might be performed before the system kernel core-dump can complete, resulting in an incomplete core-dump. For a list of valid failpolicy sets, see “Data Integrity Protection” in Chapter 1.
For example, to perform a reset only if a fencing action fails, specify the following:
failpolicy=Fence,Reset
Note: If you do not specify Shutdown and all of the other methods fail, the node attempting to deliver the CXFS kernel membership will stall delivering the membership until either the failed node attempts to re-enter the cluster or the system administrator intervenes using cms_intervene. Objects held by the failed node stall until membership finally transitions and initiates recovery. To perform a fence and an asynchronous reset, specify the following:
failpolicy=FenceReset
hostname is by the fully qualified hostname. Use the ping to display the fully qualified hostname. Do not enter an IP address. The default for hostname is the fully qualified domain name for the value of name.
nodeid is an integer in the range 1 through 32767 that is unique among the nodes in the cluster. If you change this value after a node has been defined, you must reboot the affected node. You do not normally need to specify this attribute because cxfs_admin will calculate an ID for you.
partition_id uniquely defines a partition in a partitioned Origin 3000 system, Altix 3000 series system, or Altix 4700 system. For a non-partitioned system, this attribute is not required (the default unassigned).
To unset the partition ID, use a value of 0.
For an Altix 3000, you can find the partition ID by reading the proc file. For example:
altix# cat /proc/sgi_sn/partition_id 0
The 0 indicates that the system is not partitioned. If the system is partitioned, the number of partitions (such as 1, 2, etc.) is displayed.
reset_method can be one of the following:
powerCycle shuts off power to the node and then restarts it
reset simulates the pressing of the reset button on the front of the machine (recommended)
nmi (nonmaskable interrupt) performs a core-dump of the operating system kernel, which may be useful when debugging a faulty machine
Note: NMI depends upon kernel support, which may not be present on all SGI Altix ia64 systems; if the kernel support is not provided, the nmi setting will not work.
NMI is not available on systems containing a baseboard management controller (BMC), such as SGI Altix XE x86_64 systems.
The default is powerCycle.
reset_port is the system controller port type based on the node hardware, as show in Table 11-1.
reset_password is the password for the node's system controller port (not the node's root password or PROM password). On some nodes, the system administrator may not have set this password. If you wish to set or change the system controller password, consult the hardware manual for your node.
reset_status specifies if the system reset capability is turned on (enabled) or turned off (disabled). Using disabled allows you to provide information about the system controller but temporarily disable reset (meaning that CXFS cannot reset the node). The default for nodes with system controllers is enabled, for nodes without system controllers, the default is disabled; see “Hardware and Software Requirements for Server-Capable Administration Nodes” in Chapter 1.
reset_node specifies the name of the node that is to send the reset command. It can be the logical name, hostname, or fully qualified domain name. If you use reset_comms=tty, serial cables must physically connect the node being defined and the owner node through the system controller port. The node must be defined in the cluster database.
reset_comms is tty for TTY serial devices, network for network reset to systems with L2 system controllers (you can use network reset even if you have hardware reset capability.), or ipmi for intelligent platform management interface (IPMI) network reset to systems with BMC system controllers.
reset_device is one of the following:
For systems with serial ports (reset_comms=tty ), this is the name of the terminal port (TTY) on the owner node (the node issuing the reset). A serial cable connects the terminal port on the owner node to the system controller of the node being reset. /dev/ttyd2 is the most commonly used port, except on Origin 300 and Origin 350 systems (where /dev/ttyd4 is commonly used) and Altix 350 systems (where /dev/ttyIOC0 is commonly used).
Note: Check the owner node's specific hardware configuration to verify which tty device to use. For systems with network-attached L2 system controllers (reset_comms=network), this is the IP address or hostname of the L2 controller on the node being reset. For example, reset_device=nodename-l2.mycompany.com.
For systems with network-attached BMC system controllers (reset_comms=ipmi), this is the IP address or hostname of the BMC controller on the node being reset. For example, reset_device=nodename-bmc.mycompany.com.
For example:
For an Origin 3000 series system:
reset_comms=tty reset_device=/dev/ttyd2 |
For an SGI Altix 3000 Bx2 system:
reset_comms=network reset_device=nodename-l2.mycompany.com |
For an Altix 350 system without an L2:
reset_comms=tty reset_device=/dev/ttyIOC0 |
For an Altix XE system with a BMC:
reset_comms=ipmi reset_device=nodename-bmc.mycompany.com |
Table 11-1. System Controller Types
bmc | l1 | l2 | mmsc | msc |
|---|---|---|---|---|
Any Altix XE | Origin/Onyx 300/350 | Any Altix with an L2 | Rackmount SGI 2400/2800 | Origin 200 |
| Origin/Onyx 3200C | Prism | Onyx2 | Onyx2 Deskside |
|
| Origin/Onyx 3000 series |
| SGI 2100/2200 deskside systems |
|
| Origin 300/350 |
|
|
The following is an example of a node created for BMC reset, where the default for reset_password is admin as explained in“BMC System Controller” in Appendix D (line breaks added here for readability):
cxfs_admin:mycluster> create node name=node1 type=server_admin private_net=192.168.0.168 enabled=true hostname=node1.example.com failpolicy=Fence,Reset,Shutdown nodeid=1 reset_method=powerCycle reset_port=bmc reset_status=enable reset_node=node2 reset_comms=ipmi reset_device=node1-bmc.mycompany.com |
| Note: Before deleting a server-capable administration node, you must first unmount any filesystems for which the node is a potential metadata server. |
To delete a node from the cluster and the cluster database, use the following command:
delete [node:]nodename |
If the node is enabled (which is the default), you must disable it before you delete it. For example, if mynode is a unique name in the cluster database:
cxfs_admin:mycluster> disable mynode cxfs_admin:mycluster> delete mynode |
| Note: If you delete an active metadata server, cxfs_admin will enter read-only mode. You can use the lock or lock steal=true to reenter lock mode. For more information, see “Making Changes Safely”. |
To allow a disabled node to join the cluster, enter the following:
enable [node:]nodename |
For example, if node1 is a unique name in the cluster database:
cxfs_admin:mycluster> enable node1 |
To prevent a node from joining the cluster, enter the following:
disable [node:]nodename |
For example, if node1 is a unique name in the cluster database:
cxfs_admin:mycluster> disable node1 |
| Note: This procedure is strongly recommended for stopping CXFS services on CXFS server-capable administration nodes or a client-only tie-breaker node that must be shutdown for maintenance or otherwise taken offline. These types of nodes will affect the CXFS kernel membership quorum calculation as long as they are configured as enabled in the cluster database. |
After you have disabled a node, the node is no longer an active member of the cluster.
| Caution: If you disable a node, it will be marked as Disabled and it will therefore not rejoin the cluster after a reboot. To allow a node to rejoin the cluster, you must enable the node. See “Enable a Node with cxfs_admin”. |
You can display a node's parameters with the following command:
show [node:]nodename |
For example, if cxfsxe5 is a unique name in the cluster database:
cxfs_admin:clusterOne> show cxfsxe5
Event at [ Jan 21 15:37:37 ]
node:cxfsxe5:
cellid=0
enabled=true
os=Linux
private_net:
10.0.199.1
status:
age=17
connected=true
fencing=Stable
license:
have_license=true
summary=Stable
version=5.4.0.1_ALPHA
wwns:
210000e08b123d95
type=server_admin |
| Note: The have_license field reports licenses for client-only nodes that have been obtained from the server. It does not display information about the server licenses. For more information about licensing, see “Show License Information with cxfs_admin”. |
You can see a list of all of the nodes that have been defined with the following command:
show node |
For example (output truncated):
cxfs_admin:clusterOne > show node
Event at [ Jan 21 15:54:10 ]
node:
bert:
cellid=1
enabled=false
os=Linux
private_net:
10.0.199.3
status:
age=-
connected=false
summary=Disabled
type=server_admin
cxfsxe5:
cellid=0
enabled=true
os=Linux
private_net:
10.0.199.1
status:
age=17
connected=true
fencing=Stable
license:
have_license=true
summary=Stable
version=5.4.0.1
wwns:
210000e08b123d95
type=server_admin
cxfs3:
cellid=4
enabled=false
os=IRIX
private_net:
10.0.199.33
status:
age=0
client=query
connected=true
fencing=Stable
filesystems=down
license:
allocated=0
cpu_count=2
have_license=false
oem=none
os=IRIX64
tier=
membership=down
summary=Disabled
version=5.4.0.1
wwns:
0000000000000000, 210000e08b02f8bd
xvm=down
type=client_only... |
See also “Displaying the License Keys with cxfs_admin After Installing CXFS” in Chapter 5.
| Note: These commands are only effective for the local node (the node on which cxfs_admin is running), which must be a server-capable administration node for these specific commands. |
You should revoke CXFS kernel membership of the local node only in the case of error, such as when you need to perform a forced CXFS shutdown (see “Shutdown of the Database and CXFS” in Chapter 12).
To revoke CXFS kernel membership for the local node, use the stop_cxfs command. For example:
cxfs_admin:mycluster> stop_cxfs |
The result of the stop_cxfs command is considered as a node failure by the rest of the cluster. The cluster may then fail due to a loss of CXFS kernel membership quorum or it may reset the failed node. To avoid the reset, modify the local node's definition to disable the system controller status.
To permit the local node to reapply for membership, use the start_cxfs command. For example:
cxfs_admin:mycluster> start_cxfs |
This section discusses the following:
To create the cluster, use the following command (line breaks shown here for readability, defaults in parentheses):
create cluster name=clustername Advanced-mode: heartbeat_monitor=dynamic|static (static) id=clusterID |
For example:
cxfs_admin:> create cluster name=mycluster |
You can use the modify command to add a tiebreaker node or change the CXFS kernel heartbeat monitor type. (You cannot change the cluster's name or ID.)
modify clustername tiebreaker=client_only_nodename Advanced-mode: heartbeat_monitor=dynamic|static |
For example, if mycluster is a unique name in the cluster database, to make the client-only node clientA the CXFS tiebreaker:
cxfs_admin:mycluster> modify mycluster tiebreaker=clientA |
Basic-mode usage notes:
clustername is the logical name of the cluster. It cannot begin with a number or an underscore (_), or include any whitespace characters, and can be at most 255 characters.
tiebreaker specifies the CXFS tiebreaker. See “Create a Tiebreaker with cxfs_admin”.
Advanced-mode usage notes:
heartbeat_monitor specifies how CXFS kernel membership is monitored. All nodes send CXFS kernel heartbeat messages once per second. If a node does not receive a heartbeat within a defined period, that node loses membership and is denied access to the cluster's filesystems. The defined period is one of the following:
static: Monitors constantly at 1-second intervals and declares a timeout after 5 consecutive missed seconds (default).
dynamic: Starts monitoring only when the node is processing a message from another node (such as for token recall or XVM multicast) or when the client monitors the server because it has a message pending (for example, a token acquire or metadata operation). Once monitoring initiates, it monitors at 1-second intervals and declares a timeout after 5 consecutive missed seconds, just like static monitoring. Dynamic heartbeat monitoring is appropriate for clusters that have clients with heavy workloads; using it avoids inappropriate loss of membership. However, it may take longer to recover a client's tokens and other state information when there is an actual problem.
Note: You should not use the Shutdown fail policy on client nodes if you choose dynamic heartbeat monitoring for the cluster.
id is a unique number within your network in the range 1 through 255. The cluster ID is used by the operating system kernel to make sure that it does not accept cluster information from any other cluster that may be on the network. The kernel does not use the database for communication, so it requires the cluster ID in order to verify cluster communications. This information in the kernel cannot be changed after it has been initialized; therefore, you cannot change a cluster ID after the cluster has been defined. Clusters must have unique IDs.
The CXFS tiebreaker node determines whether a CXFS kernel membership quorum is maintained when exactly half of the server-capable administration nodes can communicate with each other. There is no default CXFS tiebreaker.
| Caution: SGI recommends that you use client-only nodes as tiebreakers
to ensure that the cluster remains operational. cxfs_admin
will only let you specify a server-capable administration node as a tiebreaker
if the cluster contains four or more server-capable administration nodes,
and an even number of server-capable administration nodes.
The reset capability or I/O fencing with switches is mandatory to ensure data integrity for all nodes. Clusters should have an odd number of server-capable administration nodes. If you have an even number of server-capable administration nodes, define a CXFS tiebreaker node. |
To set the CXFS tiebreaker node, use the modify command as follows:
modify [cluster:]clustername tiebreaker=client_nodename |
For example:
cxfs_admin:mycluster> modify mycluster tiebreaker=myclient |
To unset the CXFS tiebreaker node, do not supply a value for tiebreaker. For example:
cxfs_admin:mycluster> modify mycluster tiebreaker= |
To delete a cluster, use the following command:
delete [cluster:]clustername |
For example, if mycluster is a unique name in the cluster database:
cxfs_admin:mycluster> delete mycluster |
However, you cannot delete an active cluster; you must first unmount and delete the filesystems, disable and delete the nodes, and so on.
To display the cluster, use the following command:
show cluster |
For example:
cxfs_admin:clusterOne > show cluster
Event at [ Jan 21 16:06:45 ]
cxfs:cluster:
clusterOne:
access:
admin=bert, server
monitor=
autoconf:
allowed:
pg-27
disallowed:
no-19, no-pg-27
clusterid=9
failover_net:
(none)
filesystem:
zj01s0, zj01s1, zj0ds2
heartbeat_monitor=static
node:
bert, cxfsxe5, cxfs3, penguin17, pg-27
status:
filesystems:
stable=true
summary=Stable
licenses:
cxfs_client
nodes:
stable=true
summary=Stable
stable=true
summary=stable
switch:
brocade26cp0
tiebreaker= |
The filesystem class represents the clustered XVM volumes that can be mounted by CXFS nodes. Before you can create a filesystem definition, you must create the clustered XVM volume and make the filesystem with mkfs.
By default, the filesystem:
Uses the XVM device of the same name
Enables all nodes to mount the filesystem
Mounts the filesystem in /mnt/
Is not managed by GRIOv2
To override these defaults, use the optional attributes listed below.
This section discusses the following:
Use the following commands to define a filesystem and the nodes on which it may be mounted (line breaks shown here for readability, defaults in parentheses):
create filesystem name=fsname [options=mount_options] [forced_unmount=true|false] (false) [mountpoint=mountpoint] (/mnt/fsname) [mounted=true|false] (true) Advanced-mode: [device=volumename] (fsname) [servers=server_list] (all servers are potential MDS) [nodes=nodes_that_can_mount] (all nodes can mount) [mount_new_nodes=true|false] (true) [grio_managed=true|false] (false) [grio_qual_bandwidth=qualified_bandwidth] |
You will a similar set of commands to modify a CXFS filesystem. (To modify a filesystem, you must first unmount it.)
| Note: Relocation is disabled by default. Recovery and relocation are supported only when using standby nodes. Therefore, you should only define multiple metadata servers for a given filesystem if you are using the standby node model. See “Relocation” in Chapter 1. |
Basic-mode usage notes:
name specifies the name of the filesystem:
Note: You must create XVM volumes with xvm before you set it up using cxfs_admin.
The device value must be the XVM volume name. In basic mode, you are not prompted for a device name; cxfs_admin uses the value for name and prepends /dev/cxvm/ to it. Therefore, in basic mode, the value you specify for name must be the XVM volume name. If you want to use a name value other than the XVM volume name, then you must use advanced mode and specify the device attribute.If you also specify a value for device, then name can be any string that does not begin with a number or an underscore (_), or include any whitespace characters, and can be at most 255 characters. For example, if the full XVM volume name is /dev/cxvm/concat1:
cxfs_admin:mycluster> create filesystem name=filesys1 device=concat1
If you do not specify a value for device, then name must be the name of the XVM volume name following /dev/cxvm. For example:
cxfs_admin:mycluster> create filesystem name=concat1
Note: Within the GUI, the default is to use the last portion of the device name; for example, for a device name of /dev/cxvm/d76lun0s0 , the GUI will automatically supply a logical filesystem name of d76lun0s0. The GUI will accept other logical names defined with cxfs_admin but the GUI will not allow you to modify a logical name; you must use cxfs_admin to modify the logical name. options specifies the mount options that are passed to the mount operating system command. These mount options control access to the specified filesystem. For a list of supported mount options, see the CXFS 5 Client-Only Guide for SGI InfiniteStorage. By default, this is unassigned.
Specify multiple mount options as a comma-separated list. For example, the following specifies that the myfs filesystem uses inode64 allocation and does not update the access time stamps for files and directories:
cxfs_admin:mycluster> create filesystem name=myfs options=inode64,noatime
Note: No validation is done on the mount options in cxfs_admin , so an invalid option may prevent the filesystem mounting on all nodes. forced_unmount controls the action that CXFS takes if there are processes that have open files or directories in the filesystem to be unmounted:
If set to true, the processes will be killed and the unmount will occur
If set to false, the processes will not be killed and the filesystem will unmount only after all references to the filesystem have been closed (default)
mounted specifies whether a new filesystem is mounted on all nodes in the cluster (true) or not mounted on any nodes (false). By default, the new filesystem is mounted on all nodes (true).
mountpoint specifies a mount point for the filesystem. The mount point is a directory to which the XVM volume is attached. This directory name must begin with a slash (/). The default is /mnt/fsname.
For example, to create a filesystem named myfs and use the default mount point of /mnt/myfs:
cxfs_admin:mycluster> create filesystem name=myfs
To create the myfs filesystem but use a mount point of /tmp/myfs:
cxfs_admin:mycluster> create filesystem name=myfs mountpoint=/tmp/myfs
Advanced-mode usage notes:
device is the name of the XVM volume (minus /dev/cxvm) . The default is the filesystem name specified by name.
For example, to create a device name of mydev for the myfs filesystem:
cxfs_admin:mycluster> create filesystem name=myfs device=mydev
servers specifies the potential metadata servers that can serve the filesystem to the cluster. To specify multiple server capable administration nodes, use a comma-separated list of node names. The default is all server-capable administration nodes in the cluster. The list of potential metadata servers for a given filesystem is ordered, but because of network latencies and other unpredictable delays, it is impossible to predict which node will become the active metadata server for the initial mount of a filesystem.
Note: Before deleting a server-capable administration node, you must first unmount any filesystems for which the node is a potential metadata server. For example, to specify that either node2 or node3 could be the metadata server, with node2 being the primary server, for the myfs filesystem:
cxfs_admin:mycluster> create filesystem name=myfs servers=node2,node3
nodes specifies the only nodes that can mount the filesystem as a specified comma-separated list; you must include the server-capable administration nodes in this list. If you do not specify nodes on the create command, all nodes can mount the filesystem. If you restrict the nodes on the create command line, you can later mount all nodes by specifying all of them with the nodes attribute.
For example, to restrict mounting the myfs filesystem to nodes node1 and node2:
create myfs nodes=node1,node2
To add node3:
modify myfs nodes=node1,node2,node3
mount_new_nodes specifies whether a newly created node will automatically mount the filesystem when it gets membership (true) or will not mount the filesystem (false). By default, new nodes mount all defined filesystems.
For example, to create filesystem myfs that is not automatically mounted by new nodes, use the following command:
cxfs_admin:mycluster> create filesystem name=myfs mount_new_nodes=false
To later mount the filesystem on node3 after it has been created, use the following command:
cxfs_admin:mycluster> mount myfs nodes=node3
grio_managed specifies whether a filesystem is managed by GRIOv2 (true) or not (false ) if GRIOv2 has already been enabled on the node. The default is false. Setting grio_managed to false disables GRIO management for the specified filesystem, but it does not reset the grio_qual_bandwidthvalue. In this case, grio_qual_bandwidth is left unmodified in the cluster database and ignored.
grio_qual_bandwidth specifies a filesystem's qualified bandwidth in bytes (B suffix), kilobytes (KB), megabytes (MB), or gigabytes (GB), where the units are multiples of 1024. The default is MB for 4000 or less, B for 4001 or greater. If the filesystem is GRIO-managed, you must specify a qualified bandwidth with this attribute. To modify the qualified bandwidth, the filesystem must be unmounted.
For example, the following commands all create the myfs filesystem with a GRIOv2 qualified bandwidth of 1.2 GB/s (assuming that grio_managed=true has already been set):
cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1288500000 cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1258300KB cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1288.8MB cxfs_admin:mycluster> create filesystem name=myfs grio_qual_bandwidth=1.2GB
For example, using prompting in basic mode:
cxfs_admin:mycluster> create filesystem name? myfs options? rw forced_unmount? false mountpoint? /mnt/myfs mounted? true Event at [ Jan 21 16:07:25 ] Filesystem "myfs" has been created, waiting for it to be mounted on all assigned nodes... Waiting for filesystem myfs, current status: A server is trying to mount Waiting for filesystem myfs, current status: node1 trying to mount, node2 trying to mount Waiting for filesystem myfs, current status: node1 trying to mount Operation completed successfully |
For example, using prompting in advanced mode:
cxfs_admin:mycluster> create filesystem Specify the attributes for create filesystem: name? myfs options? rw forced_unmount? false mountpoint? /mnt/myfs device? myfs servers? node1,node2,node3 nodes? node1,node2,node3,node4 mounted? true mount_new_nodes? true grio_managed? false Event at [ Jan 21 16:07:35 ] Filesystem "myfs" has been created, waiting for it to be mounted on all assigned nodes... Waiting for filesystem myfs, current status: A server is trying to mount Waiting for filesystem myfs, current status: node1 trying to mount, node2 trying to mount,node3 trying to mount, node4 trying to mount Waiting for filesystem myfs, current status: node1 trying to mount Operation completed successfully |
| Note: After a filesystem has been defined in CXFS, running mkfs on it will cause errors to appear in the system log file. To avoid these errors, run mkfs before defining the filesystem in CXFS, or delete the CXFS filesystem before running mkfs. See “Delete a CXFS Filesystem with cxfs_admin”. |
The mount command operates on the set of nodes that were specified in the nodes=nodelist attribute when the filesystem was created. By default, this is all nodes in the cluster.
To mount the filesystem on all enabled nodes in the cluster:
mount filesystem |
To mount the filesystem on specific enabled nodes:
mount filesystem nodes=nodelist |
For example, to mount the filesystem myfs on only nodes node2 and node3:
cxfs_admin:mycluster> mount myfs nodes=node2,node3 |
| Note: If any nodes that are set to mount the filesystem are enabled and attached are not in membership, mounting a filesystem will not completely succeed. The filesystem will be marked as ready to be mounted when the nodes achieve membership. |
To unmount a filesystem from all nodes in the cluster:
unmount filesystem |
To unmount the filesystem from a specific comma-separated list of nodes:
unmount filesystem nodes=nodelist |
For example, to unmount filesystem myfs from nodes node1 and node3:
cxfs_admin:mycluster> unmount myfs nodes=node1,node3 |
| Note: If any nodes are not in membership, the filesystem will be marked as not to be mounted when the nodes achieve membership. |
The relocate command forcefully moves a filesystem's metadata server to another node in the cluster that has already been defined as a potential metadata server for that filesystem. This action is typically used to free a server so it can be brought down for maintenance or upgrades. Relocation must also be explicitly enabled in the kernel with the cxfs_relocation_ok system tunable parameter (see “Relocation” in Chapter 1).
If relocation is explicitly enabled in the kernel, you can relocate a metadata server to another node by using the following command:
relocate filesystem server=new_metadata_server |
For example:
cxfs_admin:mycluster> relocate myfs server=node2 |
CXFS kernel membership is not affected by relocation. However, users may experience a degradation in filesystem performance while the metadata server is relocating.
Use the following command to delete a filesystem:
delete [filesystem:]filesystem |
You cannot delete a mounted filesystem; you must first unmount it. For example, if myfs is a unique name in the cluster database:
cxfs_admin:mycluster> unmount myfs cxfs_admin:mycluster> delete myfs |
To show information about all filesystems:
show filesystem |
To show information about a specific filesystem:
show [filesystem:]fsname |
For example, for a filesystem named zj0ds2:
cxfs_admin:clusterOne > show zj0ds2
Event at [ Jan 21 16:09:26 ]
filesystem:zj0ds2:
device=zj0ds2
forced_unmount=false
grio_managed=false
grio_qual_bandwidth=0B
mount=true
mount_new_nodes=true
mountpoint=/mnt/zj0ds2
nodes:
bert, cxfs3, cxfsxe5
options=
servers:
cxfsxe5, bert
status:
blocksize=4.00KB
free=1.44GB
nodes:
bert=disabled
cxfs3=disabled
cxfsxe5=mounted
server=cxfsxe5
size=5.58GB
stable=true
summary=Mounted
utilization=74% |
To allow the cluster to continue operation if the primary private network fails, you can set up private network failover.
To inform the servers of the failover networks, you must create a failover_net network.
| Caution: Do not create a failover_net network while CXFS services are active; doing so can lead to cluster malfunction. |
Each node in the cluster must have all private_net values specified to match the subsets defined by the failover networks in the same order as all other nodes in the cluster.
Command syntax:
create failover_net network=IPaddress mask=IPmask |
To create two private networks, one on the 192.168.0. x and the other on the 10.0.0.x subnets, use the following command:
cxfs_admin:mycluster> create failover_net network=192.168.0.0 mask=255.255.255.0 cxfs_admin:mycluster > create failover_net network=10.0.0.0 mask=255.255.255.0 |
To create a node with failover network support:
cxfs_admin:mycluster> create node name=mynode private_net=192.168.0.2,10.0.0.2 |
This section discusses the following:
For general information, see “I/O Fencing” in Chapter 2.
| Note: Nodes without system controllers require I/O fencing to protect data integrity. A switch is mandatory to support I/O fencing; therefore, multiOS CXFS clusters require a switch. See the release notes for supported switches. |
To raise or lower a fence, or update switch port information, use the hafence command.
To define a new switch, use the following command:
create switch name=switch_hostname [user=username] (admin) [password=username_password] (password) [vendor=brocade|qlogic|site-specific_vendor] (brocade) Advanced-mode: [mask=ports_that_will_not_be_fenced] |
| Note: You must define all of the switches within your fabric to which a CXFS client or server is connected. |
Basic-mode usage notes:
name specifies the hostname of the Fibre Channel switch; this is used to determine the IP address of the switch.
password specifies the password for the specified username. The default is password.
user specifies the user name to use when sending a telnet message to the switch. The default is admin.
vendor specifies the vendor of the Fibre Channel switch. It can be one of the following values:
brocade (default) qlogic site-specific-value For example, if myswitch is a QLogic switch:
cxfs_admin:mycluster> create switch name=myswitch vendor=qlogic
Advanced-mode usage notes:
mask specifies the ports on the switch that will never be fenced. By default, no ports are masked (and therefore all ports are available for fencing). The value for mask is a series of comma-separated port ranges. For example, the following states that ports 0, 4 and 12 to 15 for myswitch will never be fenced by CXFS:
cxfs_admin:mycluster> create switch name=myswitch mask=0,4,12-15
Note: For non-bladed switches, the port column of the switchShow output is the port number for CXFS.
For bladed switches, where the port number is not unique, the port column of switchShow output means the port number of one slot. However, CXFS requires a unique number identified by the combination of slot and port for bladed switch, which is usually represented by the Index or Area value in the switchShow output (depending upon the switch version). For example, the switchShow output below for the Brocade 48000 switch indicates that you would use a mask value of 16 for port 0 in slot 2:brocade48000:admin> switchShow Index Slot Port Address Media Speed State Proto =================================================== 0 1 0 010000 id N4 Online F-Port 10:00:00:00:c9:5f:9b:ea 1 1 1 010100 id N4 Online F-Port 10:00:00:00:c9:5f:ab:d9 .... 142 1 30 018e00 id N4 Online F-Port 50:06:0e:80:04:5c:0b:46 143 1 31 018f00 id N4 Online F-Port 50:06:0e:80:04:5c:0b:66 16 2 0 011000 id N4 Online F-Port 10:00:00:00:c9:5f:a1:f5 17 2 1 011100 id N4 Online F-Port 10:00:00:00:c9:5f:a1:72 ...
For more details, see the switch documentation.Server-capable administration nodes automatically discover the available HBAs and, when fencing is triggered, fence off all of the Fibre Channel HBAs when the Fence or FenceReset fail policy is selected. However, masked HBAs will not be fenced. Masking allows you to prevent the fencing of devices that are attached to the SAN but are not shared with the cluster, to ensure that they remain available regardless of CXFS status. You would want to mask HBAs used for access to tape storage, or HBAs that are only ever used to access local (nonclustered) devices.
To delete a switch, use the following command:
delete [switch:]switch_hostname |
For example, if myswitch is a unique name in the cluster database:
cxfs_admin:mycluster> delete myswitch |
To display all of the switches in the system, use the following command:
show switch [output=full_pathname] |
For example, in basic mode:
cxfs_admin:cc_test2> show switch
Event at [ Jan 21 16:12:02 ]
switch:
fcswitch12:
hostname=fcswitch12
num_ports=32
port:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
vendor=brocade
fcswitch13:
hostname=fcswitch13
num_ports=32
port:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
vendor=brocade |
To send the output to the /tmp/switchinfo file:
cxfs_admin:mycluster> show switch output=/tmp/switchinfo |
To display a specific switch:
show [switch:]switchname [output=full_pathname] |
To display mask values, use advanced mode (see “Basic and Advanced Mode”.) For example, if myswitch is a unique name in the cluster database:
cxfs_admin:cc_test2> show fcswitch12
Event at [ Jan 21 16:12:03 ]
switch:fcswitch12:
hostname=fcswitch12
mask=
num_ports=32
port:
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31
switchid=Switch1
vendor=brocade |
To display the switches and ports each host is connected to:
show wwns |
For example (output truncated):
cxfs_admin:cc_test2> show wwns
Event at [ Jan 21 16:12:04 ]
node:cc-xe:status:wwns:
100000062b0f5284:
switch=fcswitch12
switch_port=11
100000062b0f5285:
switch=fcswitch13
switch_port=11
node:cc-xe2:status:wwns:
100000062b0f568c:
switch=fcswitch13
switch_port=20
100000062b0f568d:
switch=fcswitch12
switch_port=20
node:aiden:status:wwns:
210000e08b01479d:
switch=fcswitch12
switch_port=19
210000e08b041a3a:
switch=fcswitch13
switch_port=17
...
|
To show full status details for each port on the switch, use one of the following commands:
show [switch:]switchname all show switchname:port |
For example, for the switch named fcswitch12:
cxfs_admin:mycluster> show fcswitch12:port |
This section discusses the following:
The cxfs_admin config command displays a series of commands that represent the current configuration of the objects specified. You can use this output to recreate the configuration of the entire cluster or a subset of it.
By default, config displays information at the cxfs_admin prompt. To write the configuration output to a file, use the output attribute and specify the full pathname of the file to contain the information:
config node output=full_pathname |
You can use the generated file with the -f command line option to recreate the configuration at a later time.
| Note: For a more readable configuration output (without the related commands), use the show command rather than the config command. |
To create cxfs_admin scripts, do the following:
Dump the entire cluster configuration to a file, such as /tmp/config1:
cxfs_admin:mycluster> config * output=/tmp/config1
Edit the /tmp/config1 file:
Modify the output so that the first node created is the server-capable administration node from which you are going to execute the cxfs_admin command to create the cluster. (By default, the cxfs_admin config command output lists nodes in alphabetical order by node name without regard to node type.)
Cut the create filesystem command lines at the end of the file and paste them into a second script file, such as /tmp/config2
To recreate the cluster using cxfs_admin scripts, do the following:
Recreate the base cluster (except for the CXFS filesystems) by using the first script file (/tmp/config1 ):
admin# /usr/cluster/bin/cxfs_admin -s -f /tmp/config1
Reboot the server-capable administration node:
admin# reboot
Repeat the reboot command on each server-capable administration node in the cluster.
Recreate the CXFS filesystems by using the second script file (/tmp/config2):
admin# /usr/cluster/bin/cxfs_admin -f /tmp/config2