RAIDs require routine monitoring and maintenance to ensure high reliability. This chapter describes a variety of routine administration tasks that are specific to RAID or particularly important for RAID:
Even though a RAID provides redundancy and disk drive failure prediction, it is still important to make regular tape backups of data stored on a RAID.
The IRIX Advanced Site and Server Administration Guide provides information about tape backup tools and strategies in Chapter 5, “Backing Up and Restoring Data.”
Warnings and error information for RAIDs are written to the file /var/adm/SYSLOG and the console by the raid(1M) command if the –L option is given and by the RAID device driver. (See usraid(7M) for more information about the RAID device driver.) /var/adm/SYSLOG should be checked frequently, about every 12 hours, for messages that predict disk drive failure.
An easy way to monitor /var/adm/SYSLOG automatically is to set up a cron(1M) job that checks for messages and mails the result to root or a system administrator. Follow these steps to set up a cron job:
Become superuser.
Write the current set of cron jobs to a temporary file:
crontab -l > tempfile |
Edit the temporary file and add a single line like this (shown on multiple lines here):
0 4 * * * egrep '/dev/*dsk/rad' /var/adm/SYSLOG | mail -s "RAID warnings" userid |
userid is the user ID that you want the mail sent to. At 4 a.m. every day, /var/adm/SYSLOG is searched for a string included in all RAID messages and the output is mailed to userid.
Add another line just like the one in the previous step, but change the 4 to 16:
0 16 * * * egrep '/dev/*dsk/rad' /var/adm/SYSLOG | mail -s "RAID warnings" userid |
This line performs that same check as above, but at 4 p.m. every day.
Tell cron to use the new version of tempfile:
crontab tempfile |
Another way to monitor messages in /var/adm/SYSLOG is to use syslogd(1M). See the syslogd(1M) manual page for details.
Section 5.1, “raid Error and Warning Messages,” and Section 5.2, “Messages from the RAID Device Driver,” list the messages that can appear in /var/adm/SYSLOG and describe what to do about them.
You can get configuration and status information about one or all RAIDs on a workstation or server by giving the –p option to the raid command. With this option, raid reports on disk drives that are down, RAIDs that have not been formatted, and internal configuration information in a RAID that is not consistent.
To get this information, give this command as superuser (omit the /dev/rdsk argument to check all RAIDs):
/etc/raid -p /dev/rdsk/radcontrollerdunitvh |
controller is the controller number and unit is the unit number (both can be obtained from hinv(1M)) of a RAID. An example of the output from a correctly configured RAID is:
/dev/rdsk/rad0d3vh: RAID3, sectors: 16777216, stripe size: 64, OK |
Output is written to /var/adm/SYSLOG and the console as well as being displayed in your shell window when you give the –L option on the raid command line.
Other messages that can follow the configuration information instead of OK are:
drive n is down
not initialized
must reformat
You can use the raid program to check the integrity of parity and correct errors on one RAID or all RAIDs. When checking integrity, all disk drives must be operational. Checking parity at least every few days is recommended.
To check parity on a RAID, give this command (omit the /dev/rdsk argument to check all RAIDs) as superuser:
raid -i /dev/rdsk/radcontrollerdunitvh |
controller is the controller number and unit is the unit number (both can be obtained from hinv). raid silently repairs any parity problems it finds by regenerating the parity to match the data on the disk drives. This will take 15 to 20 minutes per RAID on a lightly loaded system.
To set up a cron job that checks parity every night at 1 a.m., follow the procedure in Section 3.2, except add this line instead (shown wrapped) to tempfile:
0 1 * * * raid -i -L | mail -s "RAID warnings" userid |
The raid command with the –c option can be used to check for failed disk drives:
To check for disk drives marked down (failed disks and disks marked down with –d) on all RAIDs, give this command as superuser:
raid -c |
To check for disk drives marked down on just one RAID, give this command as superuser:
raid -c /dev/rdsk/radcontrollerdunitvh |
controller is the controller number and unit is the unit number (both can be obtained from hinv).
This command takes a few seconds. Output is written to /var/adm/SYSLOG and the console as well as being displayed in your shell window when you give the –L option, too.
When no disk drives are down, there is no output from the command. These messages are displayed if raid detects a down disk drive or other problem:
raid: warning: /dev/rdsk/radcontrollerdunitvh: drive n is down
raid: error: /dev/rdsk/radcontrollerdunitvh: transfer of config info to/from disk 2 failed: I/O error
raid: warning: /dev/rdsk/radcontrollerdunitvh: drive n is down
raid: error: /dev/rdsk/radcontrollerdunitvh: controller has been replaced
raid: error: /dev/rdsk/radcontrollerdunitvh: it doesn't match disk config info
raid: error: /dev/rdsk/radcontrollerdunitvh: disk n has been replaced, but it wasn't marked down
raid: error: /dev/rdsk/radcontrollerdunitvh: the RAID should be rebuilt
Rebuilding the RAID with raid –r makes a down disk drive operational. For more information, see Section 4.1, “Replacing a Disk Drive.”
Each time the system is rebooted, RAIDs are checked for failed disk drives automatically with this command:
raid -c -m -L |
The –m option tells the raid command to check for disk drives that were replaced while the system was off. If it finds new disks, it marks them as failed so that any data on them is not used. See Section 5.1, “raid Error and Warning Messages,” for more information.
RAID firmware can be downloaded to flash memory in the RAID controller. When IRIX software releases include new RAID firmware (see the IRIX Release Notes), you may need to download the new firmware into each RAID individually to update your system.
To update a particular RAID, first make sure that no applications are using the RAID, then give this command as superuser:
raid -l firmware /dev/rdsk/radcontrollerdunitvh |
firmware is the pathname of the file that contains the new firmware, controller is the controller number, and unit is the unit number (both can be obtained from hinv). The downloading takes about a minute.
After downloading is complete, reset the RAID controller by follow this procedure:
Open the RAID unit as explained in steps 5 and 6 of Section 4.1, “Replacing a Disk Drive.”
Press the RAID controller Reset button shown in Figure 1-5.
Close the RAID unit as explained in steps 13 and 14 of Section 4.1.