This chapter provides information about configuring the IRIS FailSafe DMF option for use on an IRIS FailSafe system. The major sections in this chapter are as follows:
The required software for IRIS FailSafe 2.1 DMF failover is as follows:
DMF software, release 2.6.3.3 or higher
See the DMF Administrator's Guide for IRIX Systems for more information about DMF.
Base IRIS FailSafe 2.1 software
If you are going to configure the DMF administrative filesystems or DMF-managed filesystems as CXFS filesystems, you must be running IRIX 6.5.13f or later, or, alternately, you must be running IRIX 6.5.10f, 6.5.11f, or 6.5.12f with patch 4264.
IRIS FailSafe DMF software, as follows:
The IRIS FailSafe Version 2 DMF Administrator's Guide subsystem is failsafe2_dmf.books.FailSafe2DMF_AG.
The IRIS FailSafe 2.1 DMF release notes subsystem is failsafe2_dmf.man.relnotes.
The base software subsystem is failsafe2_dmf.sw.base.
If files are being migrated to offline tape storage in a tape library and the tape management software being used to manage this tape storage is the IRIX Tape Management Facility (TMF), then TMF release 1.3.2.0 or later must be installed on each host; see IRIX TMF Release and Installation Guide. The IRIS FailSafe TMF agent must also be installed and configured on each host that is running DMF/TMF; for more information on the installation and configuration of the IRIS FailSafe TMF agent, see the IRIS FailSafe Version 2 TMF Administrator's Guide.
When running TMF, you must be running IRIX 6.5.12 or later. Alternately, you can run the following releases of IRIX with the indicated patches:
| IRIX 6.5.10m | patch 4210 | |
| IRIX 6.5.10f | patch 4211 | |
| IRIX 6.5.11m | patch 4187 | |
| IRIX 6.5.11f | patch 4209 |
The IRIS FailSafe DMF agent requires that you configure the following:
The DMF configuration flag
The DMF administrative filesystems and directories
The DMF-managed user filesystems
If you are using a DMF Tape MSP to migrate user files to tape, and if you will be using TMF as the tape mounting service, then you will also need to configure:
FailSafe TMF resources
These topics are discussed in the following sections.
In a FailSafe environment, DMF should be started by FailSafe itself and should not be started automatically during the boot process. After the DMF package has been installed on each node in the FailSafe cluster, execute the following command on each node to determine whether DMF has been configured to start automatically:
# /etc/chkconfig |grep dmf
dmf on
|
If the dmf value exists and is set to on, issue the following command to ensure that DMF is started only by FailSafe:
# /etc/chkconfig dmf off |
Running DMF requires a set of administrative filesystems and directories to be configured. DMF cannot start unless all of these filesystems and directories are first mounted and available. They are defined in the DMF configuration file. Table 2-1 lists the DMF administrative filesystems and directories:.
Table 2-1. DMF Administrative Filesystems and Directories
Filesystem | Attribute | Description |
|---|---|---|
Mandatory | ||
Mandatory | ||
Mandatory | ||
Optional | Used for moving files between MSPs | |
Optional | Used by a tape MSP as a cache for merging data from sparse tapes to new | |
Optional | Used by a FTP MSP to store files | |
Optional | Used by a disk MSP to store files |
In the above table, a file noted as mandatory must be defined by all users of DMF. A file noted as optional is needed only by users of certain MSP types.
In addition, the working directory used by the dmaudit command must also be available when DMF starts. The directory is configured by running the dmaudit command and selecting the <workdir> item in the <config> menu.
You can configure the DMF administrative filesystems as XLV logical volumes or as CXFS filesystems. In either case, you must configure the DMF resource to be dependent on the administrative filesystems, since the administrative filesystems must be present before DMF starts.
If you use XLV filesystems, you must define them as filesystem and volume resources and make them part of the resource group that contains the DMF resource. Failover of these resources is a fast and stable process; the filesystems are recovered quickly on a failover. The procedure for configuring the DMF administrative filesystems as XLV filesystems in a FailSafe environment is provided in “Configuring the DMF Administrative Filesystems as Shared XLV Filesystems”.
If you use CXFS filesystems, you must create a CXFS resource for each of the filesystems and add each CXFS resource to the resource group that contains the DMF resource. On failover, CXFS must recover the administrative filesystems before they are accessible. If there are many other CXFS filesystems, recovery could take many minutes. There is no way to predict or set the order of recovery and the DMF filesystems may be some of the last filesystems to be recovered. The procedure for configuring the DMF administrative filesystems as CXFS filesystems in a FailSafe environment is provided in “Configuring the DMF Administrative Filesystems as CXFS filesystems”.
To configure the DMF administrative filesystems as XLV filesystems, follow these steps:
Ensure that the DMF configuration is identical on all hosts.
Create the DMF administrative filesystems on shared disks as XLV filesystems.
Create the filesystem and volume FailSafe resources for each of those filesystems.
Add a dependency in each filesystem resource on the volume resource corresponding to that filesystem. This guarantees that FailSafe will not try to mount the filesystems until the volume labels have first been updated by the volume resource.
Create a DMF resource and create dependencies in the DMF resource on each of the filesystem resources that you created. This ensures that DMF will not start until they are all mounted.
Create the DMF resource group. For information on what a DMF resource group must include, see “Creating a DMF Resource Group”.
See “ Adding DMF Resource Information to the Configuration Database (CDB)” for more detailed information on creating a DMF resource type, a DMF resource, a DMF resource group, and a failover policy for a DMF resource group.
As indicated previously, DMF cannot start until the DMF administrative filesystems are available. If they are CXFS filesystems, CXFS must recover them before they are accessible.
To configure DMF filesystems as CXFS filesystems, follow these steps:
Ensure that the DMF configuration is identical on all hosts.
Create the DMF administrative filesystems as CXFS filesystems.
Create a CXFS resource for each filesystem. When you create a CXFS resource for a DMF administrative filesystem, be sure to configure the relocate-mds attribute to off, since DMF does not require the metadata server to fail over with the filesystem.
Create a DMF resource, and create dependencies in the DMF resource on each of the CXFS resources. This ensures that DMF will not start until these CXFS resources (filesystems) have been recovered and are available for DMF.
Create the DMF resource group. For information on what a DMF resource group must include, see “Creating a DMF Resource Group”.
See “ Adding DMF Resource Information to the Configuration Database (CDB)” for more detailed information on creating a DMF resource type, a DMF resource, a DMF resource group, and a failover policy for a DMF resource group.
As for the DMF administrative filesystems, you can configure the DMF-managed user filesystems as XLV volumes or as CXFS filesystems in a FailSafe environment. Note that for DMF-managed user filesystems, it is not necessary to create dependencies from the DMF resource to the filesystem resource since it is permissible for DMF to start before these filesystems are mounted.
To configure the DMF-managed user filesystems as XLV filesystems, follow these steps:
Create the filesystem and volume FailSafe resources for each of the filesystems.
Add a dependency in each filesystem resource on the volume resource corresponding to that filesystem. Remember to include the dmi mount option when you create each of the filesystem resources.
Add each of these resources to the resource group that contains the DMF resource.
To configure the DMF-managed user filesystems as CXFS filesystems, follow these steps:
When you create these filesystems in your CXFS cluster, make sure that you include the dmi option as a mount option for each filesystem.
Create a CXFS resource for each of these filesystems.
When you create a CXFS resource for DMF-managed user filesystem, be sure to configure the relocate-mds attribute to on. This differs from the configuration of DMF administrative CXFS filesystems, since in this case it is imperative that the metadata server fails over along with the filesystem.
Add each of these resources to the resource group that contains the DMF resource.
Whenever a CXFS FailSafe resource with the relocate-mds attribute set to on is started on a new node, it first checks to see if its filesystem is already mounted and if the CXFS metadata server for the filesystem is already running on this node. If both are true, then the filesystem is ready to use, and the CXFS resource does not have to do anything.
If either of these are not true, the CXFS resource will instigate the following actions:
The filesystem is unmounted across all nodes.
The filesystem is mounted locally on the new node, making it the new metaserver.
The filesystem is then mounted on the remaining nodes.
This sequence of actions will cause any user program which has files open within the unmounting filesystem to be killed, and the filesystem will be unavailable for a period of time until it can be remounted across all nodes.
A FailSafe administrator is free to define any type of failover policy. However, in order to minimize user disruptions an administrator may wish to define the DMF resource group failover policy in a fashion that increases the likelihood that the new FailSafe node will also be the new CXFS metaserver node, so that the above unmount/remount sequence is not necessary.
If a node on which a CXFS metadata server is running should crash, the metaserver will always move to the first available node listed in the filesystem's mnttab entry server_list parameter. This behavior matches the behavior of a FailSafe ordered failover policy with the attributes Controlled_Failback and InPlace_Recovery selected.
To minimize user disruptions in the face of server crashes, the administrator may therefore wish to define the DMF resource group to use an ordered failover policy with Controlled_Failback and InPlace_Recovery selected and enter the names of the failover domain nodes in the same order that they appear in the filesystem's server_list parameter.
If you intend to use a DMF Tape MSP to migrate user files to tape, and if you will be using TMF as the tape mounting service, then you will need to configure the following:
Define one FailSafe 2.1 TMF Agent resource for each TMF Device Group in use by DMF. For more information on the IRIS FailSafe TMF agent, see the IRIS FailSafe Version 2 TMF Administrator's Guide.
Include the TMF resources as part of the DMF resource group. This ensures that the TMF Device Group fails over along with the rest of the DMF resource group.
This section describes how to create a DMF resource type, how to create a DMF resource, and how to create a DMF resource group. It also shows how to create a failover policy for a DMF resource group. These procedures assume that a CDB that does not include DMF has already been created, installed, and tested as described in the IRIS FailSafe Version 2 Administrator's Guide.
This subsection assumes that you are already familiar with the concepts of resource types.
The DMF resource type is not automatically created at cluster creation time.The DMF resource type must be installed if you want to add a DMF resource to a cluster.
Use inst to install the DMF option.
You can use one of the following methods to create the DMF resource type:
Run cluster manager (cmgr) and manually create the resource type. For more information, see the IRIS FailSafe Version 2 Administrator's Guide.
Run cluster manager (cmgr) and install the resource type.
Instead of using the cluster manager (cmgr) to create the DMF resource type, you can use the IRIS FailSafe 2.1 GUI. Go to the “Resources & Resource Types” task page and select the “Load a Resource” task to load the DMF resource type.
The following example installs the a DMF resource type using cluster manager (cmgr).
Example 2-1. DMF Resource Type Installation
cmgr> show clusters
eagan
cmgr> show resource_types installed
NFS
template
Netscape_web
statd
Oracle_DB
MAC_address
IP_address
INFORMIX_DB
NFS
filesystem
volume
cmgr> install resource_type DMF in cluster eagan
cmgr> show resource_type DMF in cluster eagan
Name: DMF
Predefined: false
Order: 101
Restart mode: 1
Restart count: 1
Action name: stop
Executable: /var/cluster/ha/resource_types/DMF/stop
Maximum execution time: 40000ms
Monitoring interval: 0ms
Start monitoring time: 0ms
Action name: exclusive
Executable: /var/cluster/ha/resource_types/DMF/exclusive
Maximum execution time: 40000ms
Monitoring interval: 0ms
Start monitoring time: 0ms
Action name: start
Executable: /var/cluster/ha/resource_types/DMF/start
Maximum execution time: 40000ms
Monitoring interval: 0ms
Start monitoring time: 0ms
Action name: restart
Executable: /var/cluster/ha/resource_types/DMF/restart
Maximum execution time: 40000ms
Monitoring interval: 0ms
Start monitoring time: 0ms
Action name: monitor
Executable: /var/cluster/ha/resource_types/DMF/monitor
Maximum execution time: 40000ms
Monitoring interval: 180000ms
Start monitoring time: 0ms
No resource type dependencies
|
After you have defined the resource type, define a DMF resource based on the resource type. A resource requires a unique resource name (for example, the DMF resource type is the dmf instance name in the example below). To create the resource use the cluster manager (cmgr), the cmgr template scripts, or the GUI.
Example 2-2. Creating a DMF Resource Using cmgr
# /usr/cluster/bin/cluster_mgr Welcome to IRIS FailSafe Cluster Manager Command-Line Interface cmgr> define resource dmf of resource_type DMF in cluster eagan Enter commands, when finished enter either "done" or "cancel" No type specific attributes to create with set command No resource type dependencies to add resource dmf ? done Successfully defined resource dmf cmgr> |
After you have defined a resource instance of resource type DMF, you can define the failover policy for the resource group that will contain the resource.You can define any available failover policy for the resource group; however, if you are using CXFS filesystems, a failover policy of ordered with the hostnames in the failover domain defined in the same order as they are defined in the CXFS membership will result in the least impact on a user's jobs. For more information on defining a failover policy when using CXFS filesystems, see “The DMF-Managed User Filesystems”.
To determine the order of CXFS membership, execute the /etc/mount command. This will yield a list of the mounted filesystems. Check the server_list parameter of a CXFS filesystem entry to see a list of the CXFS membership in the order in which the nodes of the cluster see the membership.
To create the failover policy, use the cluster manager (cmgr), the cmgr template scripts, or the GUI. The following example shows how to define the failover policy.
Example 2-3. Defining the Failover Policy
cmgr> define failover_policy dmfadmin Enter commands, when finished enter either “done” or “cancel” failover_policy dmfadmin ? set attribute to Controlled_Failback failover_policy dmfadmin ? set script to ordered failover_policy dmfadmin ? set domain to guinness dublin failover_policy dmfadmin ? done Successfully defined failover policy dmfadmin |
To create a resource group, you must first become familiar with the terms and concepts of FailSafe. A DMF resource group should include at least the following:
A DMF resource
Resources for the DMF administrative filesystems. If these are CXFS filesystems, there must be a CXFS resource defined for each of them with the relocate-mds attribute defined as false. If these are XLV filesystems, there must be a filesystem and volume resource defined for each of them.
A resource for each DMF-managed user filesystem. If the DMF-managed user filesystems are XLV filesystems, there must be a filesystem and volume resource for each filesystem. If the DMF-managed user filesystems are CXFS filesystems, there must be a CXFS resource defined for each filesystem with the relocate-mds attribute defined as true.
If files are being migrated to offline tape storage in a tape library and the tape management software being used to manage this tape storage is the IRIX Tape Management Facility (TMF), then a resource should be defined for each TMF Device Group which is used by a DMF dmatmsp MSP process.
Set dependencies in the DMF resource for the DMF administrative filesystem resources but not for the DMF-managed user filesystems.
You must indicate what the failover policy is at the time you create the group.
The following example shows the creation of the DMF resource group dmfcxfs in cluster eagan. The DMF filesystems in this resource group are all CXFS filesystems. A CXFS resource is added to the resource group for each filesystem.
Example 2-4. Creating the Resource Group dmfcxfs of Type DMF
cmgr>
Welcome to SGI Cluster Manager Command-Line Interface
cmgr> set cluster eagan
cmgr> define resource_group dmfcxfs
Enter commands, when finished enter either "done" or "cancel"
resource_group dmfcxfs ? set failover_policy to dmfadmin
resource_group dmfcxfs ? add resource dmf of resource_type DMF
resource_group dmfcxfs ? add resource /cdmfhome of resource_type CXFS
resource_group dmfcxfs ? add resource /cdmfjournals of resource_type CXFS
resource_group dmfcxfs ? add resource /cdmfspool of resource_type CXFS
resource_group dmfcxfs ? add resource /dmf_user1 of resource_type CXFS
resource_group dmfcxfs ? add resource /dmf_user2 of resource_type CXFS
resource_group dmfcxfs ? add resource /dmf_user3 of resource_type CXFS
resource_group dmfcxfs ? add resource tmf of resource_type TMF
resource_group dmfcxfs ? done
Successfully defined resource group dmfcxfs
cmgr> show resource_group dmfcxfs
Resource Group: dmfcxfs
Cluster: eagan
Failover Policy: dmfadmin
Resources:
dmf (type: DMF)
/dmf_user1 (type: CXFS)
/dmf_user2 (type: CXFS)
/dmf_user3 (type: CXFS)
/cdmfhome (type: CXFS)
/cdmfjournals (type: CXFS)
/cdmfspool (type: CXFS)
tmf (type: TMF)
cmgr> show failover_policy dmfadmin
Failover Policy: dmfadmin
Version: 1
Script: ordered
Attributes: Controlled_Failback InPlace_Recovery
Initial AFD: guinness dublin
cmgr> show resource dmf of resource_type DMF
Resource dependencies
CXFS /cdmfhome
CXFS /cdmfjournals
CXFS /cdmfspool
cmgr> show resource /dmfuser_1 of resource_type CXFS
relocate-mds: true
No resource dependencies
cmgr> show resource /cdmfhome of resource_type CXFS
relocate-mds: false
No resource dependencies
cmgr>
|
There are several scripts that FailSafe uses to start, stop, and monitor DMF. These scripts are found in the /var/cluster/ha/resource_types/DMF directory and are as follows:
| start | Starts the DMF daemon | |
| stop | Stops DMF on this node | |
| monitor | Extended test to see if DMF is running on this node | |
| exclusive | Tests to see if DMF is already running on this node | |
| restart | Determines if the DMF daemon should be restarted |
To ensure that the DMF resource has been correctly configured, you can test individual actions by executing the scripts. Each script, located at /var/cluster/ha/resource_types/DMF, requires two arguments, an input file and an output file. The contents of the input file is the resource name. The output file is empty. The scripts will display either a 0 if they are successfully executed, or display a positive integer that indicates the error type. For more information on error codes, see the IRIS FailSafe Version 2 Programmer's Guide. The output file should be checked after testing each script to verify that the script worked properly.
For example, you can test the DMF resource dmf that was previously created by executing the following commands as root:
$ cd /var/cluster/ha/resource_types/DMF $ echo "dmf" > /tmp/ipfile |
You can then execute each script with the following command:
$ ./script-name /tmp/ipfile /tmp/opfile |
where script-name is one of the following names:
| start | Starts the DMF daemon | |
| stop | Stops DMF on this node | |
| monitor | Extended test to see if DMF is running on this node | |
| exclusive | Tests to see if DMF is already running on this node | |
| restart | Determines if the DMF daemon should be restarted |
After executing each script, verify that it worked correctly by verifying the output it generates in the /var/cluster/ha/log/script_hostname file where hostname is the name of the host on which FailSafe is running.