To replace the serial connection to the system controller port (Challenge DN/L/XL) or to the remote power control unit (Challenge S), follow these steps:

This section explains how to isolate a node (for example, node 1) with a failed component from the other node (in this case, node 2), while the system continues to run the IRIS FailSafe software. This technique allows several maintenance procedures to be applied. For example, you can replace the IO4 or the SCIP mezzanine card that affects the SCSI chains joining the two nodes.

To isolate a node (Challenge server)

---

Note: For this process, you need one differential SCSI terminator for each shared SCSI bus (Challenge RAID or vault system) to which the node is attached.

---

Disconnecting a Node From a SCSI Bus

In a dual-hosted environment, the two nodes provide termination for the SCSI bus. Since an unterminated SCSI bus causes system failures, the procedure for disconnecting SCSI buses must ensure correct termination.

For example, nodes A and B share two SCSI buses, DF SCSI4 and DF SCSI5.

• DF SCSI4 has three devices: RAID SP A, vault 1A, and vault 2A.

• DF SCSI5 has three devices: RAID SP B, vault 1B, and vault 2B.

Each volume on a vault in this example is a plex composed of a volume element from vault 1A and 1B, or vault 2A and 2B. Figure 5-1 shows this configuration.

Figure 5-1. Example Shared Storage

In this example, to isolate node B, you would

• isolate host A from SCSI bus DFSCSI-4

  • stop I/O to RAID SP A, vault 1A, and vault 2A

  • disconnect and terminate the SCSI bus at vault 2A (since it is closest to node B, which is being isolated)

  • enable I/O to RAID SP A, vault 1A, and vault 2A

• isolate host A from SCSI bus DFSCSI-5

  • stop I/O to RAID SP B, vault 1B, and vault 2B

  • disconnect and terminate the SCSI bus at vault 2B (since it is closest to node B, which is being isolated)

  • enable I/O to RAID SP B, vault 1B, and vault 2B

To disconnect the node from a SCSI bus, follow these steps:

1. Quiesce the bus by stopping I/O to all devices connected to the bus. To make this suspension of I/O transparent to applications, you must ensure that an alternate path to the data exists.

   On a Challenge RAID storage system, this alternate path exists because the LUNs can be accessed via the SP on another SCSI bus. For a Challenge Vault, the data is plexed using XLV on another Challenge Vault on a different SCSI bus.

   • To stop I/O to a vault that has been mirrored using the XLV plexing option, detach each plex on the vault from its logical volume, as explained in "Creating and Administering Volumes" in IRIX Admin: Disks and Filesystems (007-2825-001).

     ---

     Note: You must perform the detach procedure on the node that owns the XLV volume. In the IRIS FailSafe configuration, this node is the one that is currently mounting filesystems on the volume.

     ---

     
     

   • To stop I/O to an SP on the Challenge RAID storage system, disable power to the SP, as explained in Chapter 5 of the CHALLENGE RAID Installation and Maintenance Instructions (108-0128-005).

2. Disconnect the SCSI cable at the device end (Challenge RAID or vault system) of the bus closest to the node you are isolating.

3. Terminate the device with a differential SCSI terminator:

   • For a vault, remove the SCSI cable connecting the vault to the node you are isolating. Place a differential SCSI terminator on the vault's SCSI connector, as diagrammed in 5-1.

     Figure 5-2. Terminating a Vault (Isolating Node B)
     
     

For a Challenge RAID storage system, remove the SCSI cable connecting the node being isolated to the RAID SP. Attach a differential SCSI terminator to the SP's SCSI connector, as diagrammed in Figure 5-3.

Figure 5-3. Terminating a Challenge RAID Storage System (Isolating Node B)

1. Enable the alternate path that was broken in step 1 so that the system can resume activity on the SCSI bus:

   • For a plexed vault on the SCSI bus, follow the XLV procedures for attaching a plex to a logical volume, as explained in "Creating and Administering Volumes" in IRIX Admin: Disks and Filesystems (007-2825-001). Do this for each volume element on the vault.

     ---

     Caution: The plex revive operation can take up to a few hours, depending on the volume of data being revived, and the activity on the system. Do not proceed beyond this step until the revive operation has completed on all plexed volumes.

     ---

     You must perform the plex attach procedure on the node that owns the XLV volume. In the IRIS FailSafe configuration, this node is the one that is currently mounting filesystems on the volume.

     When this procedure is used for purposes other than isolating a node (for example, for replacing a nonfunctional vault disk), the node on which the revive operation should be performed might be different from the node on which the earlier detach plex procedure was performed, if a failover of filesystems occurred between these two events.

   • For a Challenge RAID device on the SCSI bus, turn on the SP to enable I/O to the bus. The SP takes about five minutes to come online. After five minutes, enter the raidcli command /usr/raid5/raidcli getcrus. The output should contain these lines to show that both SPs are online:

     SPA State: Present SPB State: Present

     
     

2. If the SCSI chain has more than one device connected to it, repeat steps 1 through 4 for each device.

Replacing a Vault Disk

---

Caution: This procedure results in the node that currently owns the volume with the failed disk to be power-cycled by the IRIS FailSafe software. For a short period, services provided by the node will not be available. We recommend that you perform this procedure during periods of light load, or during scheduled downtime.

---

To replace the disk, the entire vault must be powered off. Because powering off a vault can potentially disrupt ongoing I/O on the SCSI bus, follow these steps:

1. Identify all devices on the same SCSI bus as the faulty vault. Stop I/O to all these devices, as documented in step 1 in Section 5.2.2, "Disconnecting a Node From a SCSI Bus."

2. Use the xlv_mgr detach -force option to detach the plex containing the faulty disk.

3. Wait for HA1 to reboot. Wait for the IRIS FailSafe software to move into normal state, as displayed by ha_admin -i.

4. Power off the vault with the faulty disk, replace the disk, and power on the vault. Wait for all the disks to spin up.

5. Remake the plex object that contained the faulty disks, using xlv_make as documented in IRIX Admin: Disks and Filesystems.

6. Enable I/O to all devices on the SCSI bus as documented in step 4 in Section 5.2.2, "Disconnecting a Node From a SCSI Bus." This operation requires reattaching all detached plexes from step 1 to their respective subvolumes.

This section reproduces information from the CHALLENGE RAID Installation and Maintenance Instructions for convenience. For complete instructions on replacing field-replaceable units, see that manual.

---

Caution: Before you follow procedures explained in this chapter, note the following points:

---

Figure 5-4 diagrams location of disks, which can be replaced by owners or SSEs.

Figure 5-4. Location of Disks (Front of Challenge RAID)

---

Note: For information on replacing other FRUs (field-replaceable units) in the Challenge RAID storage system, see the CHALLENGE RAID Installation and Maintenance Guide. For information on replacing SCSI-2 adapters in the Challenge server, see the Challenge server documentation.

---

This section explains

Opening and Closing the Fan Module

To remove or install an SP, power supply, or battery backup unit, you must unlock and swing open the fan module:

1. On the back of the storage system, move the fan module's latch to the UNLOCK position, as shown in Figure 5-5.

   Figure 5-5. Unlocking the Fan Module
   
   

2. Swing open the fan module, as shown in Figure 5-6.

   Figure 5-6. Opening the Fan Module
   
   

3. Close the fan module by reversing these steps.

---

Caution: To prevent thermal shutdown of the storage system, never operate the storage system for more than two minutes with the fan module in the open position.

---

Replacing or Adding a Disk Module

This section explains

• identifying a failed disk module

• removing the failed disk module

• installing the replacement disk module

• updating the disk module firmware

Identifying and Verifying a Failed Disk Module

If you have determined that a module has failed by examining the cabinet fault light or by using the raidcli getdisk or raidcli getcrus command, you can replace the defective module and rebuild your data without powering off the Challenge RAID storage system or interrupting user applications.

---

Caution: Removing the wrong drive module can introduce an additional fault that shuts down the physical disk containing the failed module. Before removing a disk module, verify that the suspected module has actually failed.

---

The fault indicator on a disk module does not necessarily mean that the drive itself has failed. Failure of a SCSI bus, for example, lights the fault indicator on each disk module on that bus.

To verify a suspected disk module failure, follow these steps:

1. Look for the module with its amber fault light on. Figure 5-7 shows the fault indicator light and other lights on a disk module.

   Figure 5-7. Disk Module Status Lights
   
   

2. Determine the failed module's ID; see Figure 5-4.

   ---

   Caution: Use only Challenge RAID disk modules to replace failed disk modules. Order them from the Silicon Graphics hotline. Challenge RAID disk modules contain proprietary firmware that the storage system requires for correct functioning. Using any other disks, including those from other Silicon Graphics systems, can cause failure of the storage system.

   ---

   
   

3. If you have not already checked the module status with raidcli getdisk, do so now.

4. If you have not already checked the unsolicited error log with raidcli getlog for a message about the disk module, do so now.

   A message about the disk module contains its module ID (such as A0 or B3). Check for any other messages that indicate a related failure, such as failure of a SCSI bus or a general shutdown of a chassis, which might mean the disk module itself has not failed.

   ---

   Note: If you are using storage system caching, the system uses modules A0, B0, C0, D0, and E0 for its cache vault. If one of these modules fails, the storage system dumps its cache image to the remaining modules in the vault; then it writes all dirty (modified) pages to disk and disables caching. The cache status changes, as indicated in the output of the raidcli getcache command. Caching remains disabled until you insert a replacement module and the storage system rebuilds the module into the physical disk unit. For information on caching, see Chapter 7 of the CHALLENGE RAID Owner's Guide.

   ---

   
   

---

Caution: Although you can remove a disk module without damaging the disk data, do this only when the disk module has actually failed. Never remove a disk module unless absolutely necessary, and only when you have its replacement available. Never replace more than one disk module at a time; use only correct disk modules available from Silicon Graphics, Inc.

---

Removing a Failed Disk Module

You can replace a failed disk module while the storage system is powered on. If necessary, you can also replace a disk module that has not failed, such as a module that has reported many "soft" errors. When replacing a module that has not failed, you must do so while the storage system is powered on so that the storage-control processor knows the module is being replaced.

---

Caution: To maintain proper cooling in the storage system, never remove a disk module until you are ready to install a replacement. Never remove more than one disk module at a time.

---

To remove a disk module, follow these steps:

1. Verify that the suspected module has actually failed; see Section 5.3.2.1, "Identifying and Verifying a Failed Disk Module."

2. When a disk fails, the storage-control processor automatically unbinds it. To check that the disk has been unbound, use the following raidcli command for the disk module position in question:

   raidcli -d device getdisk diskposition

   If necessary, get the device name first with raidcli getagent.

3. Locate the disk module that you want to remove; see Figure 5-4 if necessary.

4. Position the new disk module in its antistatic packaging within reach of the storage system.

5. If you are using an ESD wrist strap, attach its clip to the ESD bracket at the bottom of the storage system, as shown in Figure 5-8. Put the wrist band around your wrist with the metal button against your skin.

   Figure 5-8. Attaching the ESD Clip to the ESD Bracket on the Storage System
   
   

   Figure 5-9 shows where to attach the clip on a rack storage system.

   Figure 5-9. Attaching the ESD Clip to the ESD Bracket on a Rack Storage System
   
   

6. Make sure the disk has stopped spinning and the heads have unloaded.

7. Grasp the disk module by its handle and pull it partway out of the cabinet, as shown in Figure 5-10.

   Figure 5-10. Pulling Out a Disk Module
   
   

   ---

   Caution: Never remove more than one disk module at a time.

   ---

   
   

   ---

   Warning: When removing a disk module from an upper chassis assembly in a Challenge RAID rack system, make sure that you adequately balance the weight of the disk module.

   ---

   
   

8. Supporting the disk module with your free hand, pull it all the way out of the cabinet, as shown in Figure 5-11.

   Figure 5-11. Removing a Disk Module
   
   

   ---

   Caution: When removed from the chassis, the disk modules are extremely sensitive to shock and vibration. Even a slight jar can severely damage them.

   ---

   
   

9. If necessary, on the label on the side of the disk module, write the ID number for the compartment from which you removed the drive; for example, A3.

   For the compartment ID numbers, refer to Figure 5-4.

10. Put the failed disk module in an antistatic bag and store it in a place where it will not be damaged.

Installing a Replacement Disk Module

To install the replacement disk module, follow these steps:

---

Caution: The disk module is extremely sensitive to shock and vibration. Even a slight jar can severely damage it.

---

1. Before installing a replacement module, wait at least 15 seconds after removing the failed module to allow the processor time to recognize that the module has been removed. If you insert the replacement module too soon, the processor may report the replacement module as defective.

2. Position the new disk module in its antistatic packaging within reach of the storage system.

3. Locate the slot where you will install the disk module; see Figure 5-4.

4. Touch the new disk module's antistatic packaging to discharge it and the drive module. Remove the new disk module from its packaging.

5. Engage the disk module's rail in the chassis rail slot, as shown in Figure 5-12.

   Figure 5-12. Engaging the Disk Module Rail
   
   

6. Engage the disk module's guide in the chassis guide slot, as shown in Figure 5-13.

   Figure 5-13. Engaging the Disk Module Guide
   
   

7. Insert the disk module, as shown in Figure 5-14. Make sure it is completely seated in the slot.

   Figure 5-14. Inserting the Replacement Disk Module
   
   

8. Remove and store the ESD wrist band, if you are using one.

9. The storage-control processor formats and checks the new module, and then begins to reconstruct the data. While rebuilding occurs, you have uninterrupted access to information on the physical disk unit.

The default rebuild period is four hours.

---

Note: During the rebuild period, performance might degrade slightly, depending on the rebuild time specified and on I/O bus activity.

---

For more information on changing the default rebuild period, use raidcli bind, as explained in Appendix B of the CHALLENGE RAID Installation and Maintenance Instructions.

Updating the Disk Module Firmware

After replacing a failed unbound disk module (A0, B0, C0, or A3), update the firmware on the Challenge RAID SP. The firmware (FLARE code) is stored in separate subdirectories in /usr/raid5/flare:

• /usr/raid5/flare/sauna7/flarecode.bin: use this for AMD-based SPs with FLARE code rev. level 7.xx

• /usr/raid5/flare/sauna8/flarecode.bin: use this for AMD-based SPs with FLARE code rev. level 8.00 through 8.49

• /usr/raid5/flare/phoenix8/flarecode.bin: use this for PowerPC-based SPs with FLARE code rev. level 8.50 through 8.99

• /usr/raid5/flare/phoenix9/flarecode.bin: use this for PowerPC-based SPs with FLARE code rev. level 9.00 through 9.99

Follow these steps:

1. Quiesce the bus, disabling all applications. Make sure that only the RAID agent is running.

2. Enter as root:

   raidcli -d device firmware /usr/raid5/flare/directory/flarecode.bin

   
   

In this string, the new firmware image /usr/raid5/flare/<directory>/flarecode.bin contains microcode that runs on the SP and also a microcode image destined for the SP PROM, which runs the power-on diagnostics.

You must use this command every time you replace a failed unbound disk module (A0, B0, C0, or A3).

The image in the file given in the command contains microcode that runs on the storage-control processor and possibly also a microcode image destined for the storage-control processor PROM, which runs the power-on diagnostics.

---

Note: Once the microcode has been downloaded, each SP in the cabinet reboots. The reboot process takes several minutes, during which time no command-line interface commands are accepted other than getagent. When the code is downloaded, the system reboots the SPs. After the download is complete, stop and restart the agent.

---

This command has no output. Use raidcli getagent or getsp to return the new firmware version.

Installing an Add-On Disk Module Array

When you install an add-on disk module, use only Challenge RAID disk modules; only they contain the correct device firmware. Other disk modules, even those from other Silicon Graphics equipment, do not work. Do not mix disk modules of different capacities within one array. Do not remove disk modules from bus 0 (slots A0, B0, C0, D0, and E0) for use in other disk module positions.

Table 5-1 gives marketing codes for add-on disk module arrays. Add-on disk modules are available only in arrays of five.

Table 5-1. Ordering Add-On Disk Module Sets

Unit	Marketing Code
Add-on five 2 GB drives	P-S-RAID-5X2
Add-on five 4.3 GB drives	P-S-RAID-5X4
Base array with five 2 GB drives	P-S-RAID-B5X2
Base array with five 4 GB drives	P-S-RAID-B5X4

Fill the slots in this order (consult Figure 5-4 if necessary):

• first, in this order: A0, B0, C0, D0, E0

• next, in this order: A1, B1, C1, D1, E1

• next, in this order: A2, B2, C2, D2, E2

• next, in this order: A3, B3, C3, D3, E3

---

Caution: Never remove more than one disk module or disk filler module at a time.

---

1. Position the new disk modules in their antistatic packaging within reach of the storage system.

2. If you are using a wrist band, attach its clip to the ESD bracket on the bottom of the storage system, as shown in Figure 5-8. Put the wrist band around your wrist with the metal button against your skin.

3. Locate the slots where you will install the add-on disk modules; see Figure 5-4.

4. Grasp the filler module for the first slot and pull it out of the cabinet; set it aside. If you cannot grasp the module, use a medium-size flat-blade screwdriver to pry it out gently. See Figure 5-15.

   Figure 5-15. Removing a Disk Filler Module
   
   

5. Touch the new disk module's antistatic packaging to discharge it and the drive module. Remove the new disk module from its packaging.

6. Engage the disk module's rail in the chassis rail slot, as shown in Figure 5-16.

   ---

   Caution: Disk modules are extremely sensitive to shock and vibration. Even a slight jar can severely damage them.

   ---

   Figure 5-16. Engaging the Disk Module Rail
   
   

7. Engage the disk module's guide in the chassis guide slot, as shown in Figure 5-17.

   Figure 5-17. Engaging the Disk Module Guide
   
   

8. Insert the disk module, as shown in Figure 5-17. Make sure it is completely seated in the slot.

As part of a SCSI component (SPs, cables, SCIP mezzanine cards, IO4 boards) replacement procedure, the RAID system's SPs must be powered off. At this time, accesses to the LUNs currently owned by the disabled SP are normally no longer possible.

When the SPs are turned back on (at the end of the replacement procedure), all LUNs continue to be owned by the other SP. This section explains how to restore the original status.

Certain patches upgrade IRIX so that it contains a failover feature:

If accesses to the LUNs on the RAID system are made through XLV volumes, and an SP fails (or is turned off), XLV code forces the remaining SP to take over ownership of all LUNs and redirects the accesses to the remaining SP.

This failover is transparent to the user (such as the Oracle RDBMS) making the accesses, provided the following are true:

To check that all volumes have been assembled correctly, run xlv_make, and at the prompt, enter show. If each volume element of each volume shows for each disk partition a second one in curly braces ({}), the volume has been assembled correctly.

The most common cause for incorrectly assembled volumes (barring hardware failures) is that the RAID agent was running at the time xlv_assemble was invoked. If both SPs are still operational, this problem can be rectified as follows:

However, when the original SP is turned back on (at the end of the replacement procedure), all LUNs continue to be owned by the other SP. While this situation is viable, it is not optimal. Performance is significantly improved if accesses to the LUNs are balanced across the two SPs, instead of all being funneled through one SP.

To restore the original status, use the raidcli trespass command (RAID agent 1.55; for earlier versions, use /sbin/trespass) as follows. For example, if SP A was turned off and is now operational again, and SP A is connected to SCSI bus 6 with target (SCSI ID) 6, you would type:

/usr/raid5/raidcli trespass sc6d610 mine

This command does not require the RAID agent to be stopped. However, for SPs with RAID agent earlier than 1.55, use /sbin/trespass as follows:

/etc/init.d/raid5 stop

/sbin/trespass sc6d610 mine
/etc/init.d/raid5 start

For /sbin/trespass, the first command is required because the /sbin/trespass command cannot work while the agent is running; the last command restarts the RAID agent. The /sbin/trespass or raidcli trespass command tells SP A to take over the LUNs of which it is default owner. See Appendix B of the CHALLENGE RAID Installation and Maintenance Instructions (007-0128-006) for more information on raidcli trespass; /sbin/trespass has identical options.

Follow this procedure whenever an SP (storage-control processor) of a dual-SP RAID system is turned off, then turned back on again.

The remote power control unit accepts inlet power from a standard 12 VDC-AC wall adapter.

The remote power control unit uses eight AA alkaline batteries as an on-board uninterruptable power supply to power the unit and sustain it for up to five hours during a power outage. The batteries are in a removable battery tray, which is accessed from the front of the unit.

An LED on the battery tray alerts you that the batteries have lost sufficient power to drive the remote power control unit. This LED also lights up immediately after you power on the remote power control unit.

If the batteries fail and are not replaced, the unit retains all configuration settings in the absence of power for ten years, although it cannot control the power control units.

You can replace batteries in the remote power control unit with the unit powered on and the IRIS FailSafe cluster running (the battery tray is hot-pluggable).

To install fresh batteries, follow these steps: