Chapter 2. Setting Up and Cabling the Oracle Parallel Server Component Systems


Note: Before installing an Oracle Parallel Server (OPS), make sure that the installation site meets the operating limits and AC power requirements as explained in Chapters 3 and 4 of the CHALLENGE/Onyx Site Preparation Guide.

This chapter explains how to cable the two Challenge servers (OPS hosts), the two Challenge vaults, the Challenge RAID storage system, and the IRISconsole for use as the hardware for the Oracle Parallel Server. The process consists of

  1. setting up the component systems

  2. upgrading the Remote System Control firmware

  3. upgrading the IO4 board mezzanine cards

  4. planning the cabling

  5. cabling the vaults to the OPS hosts

  6. setting the OPS host SCSI IDs

  7. installing the FDDI boards in the OPS hosts

  8. cabling the Challenge RAID storage system to the Challenge servers

  9. cabling the IRISconsole multiplexer to the workstation

  10. cabling the IRISconsole multiplexer and the Challenge servers

  11. powering on the systems

  12. testing the installed OPS hardware

The following equipment is required for installation:


Note: Make sure that each Challenge L server used for OPS has a Remote System Control port at the extreme right. If it does not, the customer must order an upgrade (013-0624-003) and you must install this port following instructions in the Remote System Control Port Installation Guide, included in this shipment.


Setting Up the Component Systems

Read through the CHALLENGE/Onyx Site Preparation Guide before unpacking equipment at the site, paying particular attention to the following:

  1. Make sure that the installation site meets the operating limits and AC power requirements as explained in Chapters 3 and 4 of the CHALLENGE/Onyx Site Preparation Guide.

  2. Make sure required tools and personnel are on hand for unloading and opening crates and for moving large systems.

  3. Prepare the physical location to allow for space, air flow, and floor-loading requirements for all component systems, as explained in Chapter 3 of the CHALLENGE/Onyx Site Preparation Guide.

    Plan to situate the OPS component systems fairly close together. Differential SCSI cables, including cabling inside the chassis, can be no longer than 25 meters (82 feet). Use only the cables included in the shipment.

  4. Make sure the site meets safety and operating considerations, as explained in Chapter 3 of the CHALLENGE/Onyx Site Preparation Guide. Note that the fully loaded Challenge RAID rack weighs 899.4 lb (407.96 kg); make sure flooring is adequate.

  5. Prepare the site for the systems' electrical requirements, as explained in Chapter 4 of the CHALLENGE/Onyx Site Preparation Guide.

  6. Connect your laptop or ASCII terminal and keyboard to the first Challenge server, as explained in the server installation instructions.

  7. Reserve the label sheets that are included with the Challenge vault systems; you need them for labeling SCSI cables for OPS.

  8. Set the SCSI bus selectors on each optional vault to 8 through 15. Figure 2-1 shows the location of the settings.

    Figure 2-1. SCSI Bus Selectors on Front of Vault

    Figure 2-1 SCSI Bus Selectors on Front of Vault

  9. Connect the power cord for the first OPS host (OPS1). Turn on the main power switch on the back of the unit; turn on the laptop or ASCII terminal. If necessary, see the server installation instructions for details.

  10. On the OPS host, turn the System Controller key switch to the On position. When power-on diagnostics are completed, a message appears.

  11. Type hinv on each OPS host; the screen should display output like the following (some lines omitted for clarity):

    6 150 MHZ IP19 Processors

    ... 
    Integral Ethernet controller: et0, Ebus slot 5
    FDDI ... : fxp0 
    EPC external interrupts
    Integral SCSI controller 1: Version WD33C95A, differential, revision 0
    Disk drive: unit 1 on SCSI controller 1
    Integral SCSI controller 0: Version WD33C95A, single ended, revision 0
    Tape drive: unit 6 on SCSI controller 0: DAT
    Integral SCSI controller 7: Version SCIP/WD33C95A
    Integral SCSI controller 6: Version SCIP/WD33C95A
    Integral SCSI controller 5: Version SCIP/WD33C95A
    Integral SCSI controller 4: Version SCIP/WD33C95A
    Integral SCSI controller 3: Version SCIP/WD33C95A
    Integral SCSI controller 2: Version SCIP/WD33C95A
    Integral EPC parallel port: Ebus slot 5
    VME bus: adapter 0 mapped to adapter 21
    VME bus: adapter 21

  12. Set up each system on the network for the site.

Upgrading the Report System Control Firmware

If necessary, install the Remote System Control port upgrade in one or both Challenge L systems, following instructions in the Remote System Control Port Installation Guide included in this shipment.

A new system controller firmware PROM is included in the OPS shipment. Install it as explained in the Remote System Control Port Installation Guide (0108-0140-001).

Upgrading the IO4 Board Mezzanine Cards

Two SCIP mezzanine cards are included in the OPS option. Install them following instructions in the CHALLENGE/Onyx XL Rackmount Installation Instructions or CHALLENGE/Onyx L Deskside Installation Instructions.

Planning the Cabling

The numbering of SCSI controllers (ports) on the OPS host IO4 board and mezzanine card(s) (daughter boards), if present, is standardized. Figure 2-2 diagrams the numbering scheme. Note that the numbering is constant regardless of which daughter board is installed.

Figure 2-2. SCSI Controller Numbering on IO4 and Mezzanine Boards

Figure 2-2 SCSI Controller Numbering on IO4 and Mezzanine Boards

In an OPS configuration, each SCSI bus runs from a controller on one Challenge server through a vault or Challenge RAID to the second OPS host. For clarity, connect the controllers with the same numbers on a bus; for example, connect cable SCSI controller 2 on the first OPS host (OPS1) to SCSI controller 2 on the second OPS host (OPS2). Figure 2-3 diagrams a typical OPS configuration.


Note: Make sure that controllers 2 and 5 and bus A on both optional vaults are differential.

Figure 2-3. Example OPS Cabling Scheme

Figure 2-3 Example OPS Cabling Scheme

Figure 2-4 shows servers and vaults cabled for OPS.

Figure 2-4. OPS Cabling for Servers and Vaults

Figure 2-4 OPS Cabling for Servers and Vaults


Note: The IRISconsole is also linked to the Challenge RAID storage system; this link is not shown in Figure 2-4.

The SCSI buses in the vaults and the Challenge RAID storage system are named A and B. The recommended procedure is to connect even-numbered SCSI buses to A and odd-numbered buses to B.

Cabling the Vaults to the OPS Hosts

This section explains how to cable each SCSI bus on each vault to each OPS host. Follow these steps:

  1. Have ready eight of the OPS host-vault cables. Use only these cables for this connection.

  2. On blank labels from the label sheet, create labels OPS1, OPS2, Vault 1, and Vault 2. Attach the labels to the appropriate OPS hosts and vaults.

  3. Power off both servers and vaults.

  4. On the back of vault 1, insert one of the cables into the top SCSI channel A socket, as shown in Figure 2-5.

  5. Label the cable end or connector; follow the scheme in Figure 2-3.

  6. Connect the other end of this cable to an even-numbered SCSI bus port on the first OPS host (OPS1), as shown in Figure 2-5. Label the connector or cable end.

    Figure 2-5. Connecting the First Challenge Vault and Server

    Figure 2-5 Connecting the First Challenge Vault and Server

  7. On the back of vault 1, insert a cable into the top SCSI channel B socket. Label the cable end.

  8. Connect the other end of this cable to an odd-numbered SCSI bus port on the first OPS host (OPS1), as shown in Figure 2-6. Label the cable end or connector.

    Figure 2-6. Connecting SCSI Buses A and B from the First Vault to the First Server

    Figure 2-6 Connecting SCSI Buses A and B from the First Vault to the First Server

  9. Insert a third cable into the lower SCSI channel A port on the back of vault 1. Label the cable end.

  10. Connect the other end of this cable to the same even-numbered SCSI bus port on the second OPS host (OPS2) that you used on OPS1, as shown in Figure 2-7. Label the cable end.

  11. Insert a fourth cable into the lower SCSI channel B port on the back of vault 1 and label it. Connect its other end to same odd-numbered SCSI bus port on the second OPS host (OPS2) that you used on OPS1. Label the cable end. Figure 2-7 shows all cables for the first vault installed.

    Figure 2-7. Cabling the First Vault to Both Servers

    Figure 2-7 Cabling the First Vault to Both Servers

  12. Connect the four remaining cables to the second vault; follow the pattern in steps 4 through 11. Figure 2-4 earlier in this chapter shows this cabling.

  13. If necessary, reset the SCSI bus selector settings on the front of each Challenge vault if they have been unset accidentally.

  14. If one or more optional Digital Linear Tape drives (DLT) were ordered, install them in the server(s).

  15. Power on the vaults. Power on the servers.

  16. Do a dummy I/O to test each link. For example, in the numbering scheme used so far, to test SCSI bus 2, send data from OPS1 to disk 8 on SCSI bus 2 (dks2d8) and see if the disk light for disk 8 on the first vault lights up. Send data from OPS2 and see if the same LED is activated. Repeat this process for each bus.

  17. Halt and power off each system.

Setting the OPS Host SCSI IDs

SCSI host address conflicts have caused the most problems in getting OPS up and running.


Caution: The two Challenge servers must have different SCSI IDs. The recommended addresses for OPS are 0 for the first OPS host (OPS1)and 2 for the second OPS host (OPS2). No SCSI device on any bus on any server should have SCSI ID 2.

Setting the host SCSI IDs consists of these steps:

  1. For the first OPS host, enter the command nvram. The last line of this command's output should contain the line

    scsihostid=

    This output means that the SCSI host ID for this OPS host is 0, the default.

  2. Repeat step 2 in the second window for the second OPS host. The last line of the output should be scsihostid=2, meaning that the SCSI host ID for the second OPS host is 2.

If the second OPS host's SCSI host ID is not 2 (probably the case), follow these steps:

  1. Reboot and start the System Maintenance menu; choose item 5, the Command Monitor.

  2. In the Command Monitor, set the SCSI ID for this system:

    setenv scsihostid 2


    Caution: No SCSI device on any bus on any server should have SCSI ID 2.


  3. To verify that the SCSI ID was set, enter

    printenv

    The output should include the line scsihostid=2.

  4. Exit the System Maintenance menu and restart the system.

  5. Enter the command hinv; see Section 2.1, "Setting Up the Component Systems," for a sample output.

  6. Enter the command hinv for the second server. Compare the output for each server.

Installing the FDDI Boards in the OPS Hosts

Have the following manuals at hand:

  • FDDIXpress Release Notes

  • FDDIXpress Administration Guide (document number 007-0813-040)

  • FDDIXpress Mezzanine Board for CHALLENGE and Onyx Installation Instructions (108-0116-001)


Caution: This equipment is extremely sensitive and susceptible to damage by electrostatic discharge (ESD), a spark caused by the buildup of electrical static potential on clothing and other material. You must use proper ESD preventive measures.

To install the FDDI boards in the servers, follow these steps:

  1. Check the contents of the FDDI board shipment, as explained in Chapter 1 of the FDDIXpress Mezzanine Board Installation Instructions.

  2. Use the versions command to see if the system into which you are installing the FDDIXPress™ mezzanine board is already running FDDIXPress software. If FDDIXPress is not installed, install it following the instructions in the FDDIXpress Release Notes.

  3. Install the FDDIXpress board following instructions in Sections 2.1 and 2.2 in Chapter 2 (Challenge L) or Sections 3.1 and 3.2 in Chapter 3 (Challenge XL) of the FDDIXpress Mezzanine Board for CHALLENGE and Onyx Installation Instructions.

  4. Cable the connections and complete the installation as explained in Chapter 4 of the FDDIXpress Mezzanine Board for CHALLENGE and Onyx Installation Instructions.

  5. Edit /etc/config/netif.options to include network addresses, for example:

     #  To override the name and/or address of the first gateway interface,
     #  change the value part and remove the leading : character.
    
     : if2name=ipg0
     : if2addr=dlm-$HOSTNAME
    
     #  If this host has more than 2 interfaces, you must define values for
     #  if3name (and if4name if appropriate). Change if3addr (and if4addr) to
     #  the appropriate names in /etc/hosts if your site has different naming
     #  conventions.
    
     if3name=
     if3addr=gate2-$HOSTNAME
    
     if4name=
     if4addr=gate3-$HOSTNAME

  6. Edit /etc/hosts to include IP numbers for OPS1 and OPS2. Use test numbers that are not in use on the customer's network. For example:

     127.0.0.1       localhost
     #
     # This network number, '192.0.2' is the officially blessed 'test' network.
     #      It should be changed immediately to your own official network number.
     
     # start of well known machines--do not remove this line
     # main ethernet network
     192.26.79.38    ops1
     192.26.79.39    ops2
     # for DLM FDDI connection only
     192.0.2.200     dlm-ops1
     192.0.2.201     dlm-ops2
     #

  7. Finish configuring the new FDDI station, as explained in Chapter 2 of the FDDIXpress Administration Guide. If you are familiar with the procedures, follow the section "Quick and Easy Configuration Instructions" in that chapter.

  8. Verify the FDDI connection, as explained in Chapter 2 of the FDDIXpress Administration Guide.

Cabling the Challenge RAID Storage System
to the Challenge Servers

The SCSI ports on the Challenge RAID storage system are cabled to both Challenge servers; consult the diagram in Figure 2-3.

Table 2-1 charts SCSI ID switch settings for the Challenge RAID storage system. Note that the settings do not conform to frequently used numbering schemes.

Table 2-1. Challenge RAID SCSI ID Switch Settings

SCSI ID Number

Switch Number

 

 

 

Comment

 

ID 0

ID 1

ID 2

ID 3

 

1

Off

On

On

On

 

2

On

Off

On

On

Do not use

3

Off

Off

On

On

 

4

On

On

Off

On

 

5

Off

On

Off

On

 

6

On

Off

Off

On

Shown in Figure 2-3 as A

7

Off

Off

Off

On

Shown in Figure 2-3 as B

8

On

On

On

Off

 

9

Off

On

On

Off

 

10

On

Off

On

Off

 

11

Off

Off

On

Off

 

12

On

On

Off

Off

 

13

Off

On

Off

Off

 

14

On

Off

Off

Off

 

15

Off

Off

Off

Off

 

To cable the Challenge RAID storage system to the Challenge servers, follow these steps:

  1. Have ready the two remaining SCSI cables and the two 20-foot SCSI cables included in the Challenge RAID shipment (p/n 9290111).

  2. Using blank labels from a label sheet, create two labels that say DF SCSI n and two labels that say DF SCSI N, where n and N are the numbers of two remaining SCSI controllers on the servers (in the example in Figure 2-3, these are 6 and 7).

  3. Attach the two even-numbered labels below the SCSI A in symbol and SCSI A out symbol silk-screens on the Challenge RAID storage system. Attach the two odd-numbered labels below the SCSI B in symbol and SCSI B out symbol on the Challenge RAID storage system. Figure 2-8 shows the Challenge RAID SCSI ports.

    Figure 2-8. SCSI–2 Bus Connectors on Back of Challenge RAID Chassis

    Figure 2-8 SCSI–2 Bus Connectors on Back of Challenge RAID Chassis

  4. On the back of the Challenge RAID storage system, attach a SCSI cable to the Challenge RAID SCSI A in port. Connect the SCSI cable to the SCSI A in connector as shown in Figure 2-9, making sure the connector is inserted securely.

    Figure 2-9. Connecting a SCSI Bus Cable to a Challenge RAID SCSI Port

    Figure 2-9 Connecting a SCSI Bus Cable to a Challenge RAID SCSI Port

  5. Connect the other end of this cable to the even-numbered SCSI bus on OPS1, as shown in Figure 2-10.

    Figure 2-10. Connecting a Challenge RAID SCSI Bus Cable to a Challenge SCSI Port

    Figure 2-10 Connecting a Challenge RAID SCSI Bus Cable to a Challenge SCSI Port

  6. Continue cabling as diagrammed in Figure 2-11.

    Figure 2-11. Challenge RAID OPS Connection Schemes

    Figure 2-11 Challenge RAID OPS Connection Schemes

    Figure 2-12 shows a typical hookup for an OPS configuration with one Challenge RAID storage system.

    Figure 2-12. Connecting SCSI Cables to a Challenge RAID and Both OPS Hosts

    Figure 2-12 Connecting SCSI Cables to a Challenge RAID and Both OPS Hosts

Cabling the IRISconsole Multiplexer to the Workstation

To cable the Indy, Indigo, or Indigo2 workstation to the Challenge servers, follow these steps:

  1. Have ready the contents of the IRISconsole multiplexer kit and the pair of cables included in the cable kit, so that you have four 2-ft DB25-DB9 serial cables (p/n018-0527-001).

  2. Connect one end of the SCSI cable (018-0526-001) included in the multiplexer kit to the workstation's SCSI port, as shown in Figure 2-13, Figure 2-14, and Figure 2-15.

    Figure 2-13. Cabling the Multiplexer to an Indy Workstation

    Figure 2-13 Cabling the Multiplexer to an Indy Workstation

    Figure 2-14. Connecting the Multiplexer's SCSI Cable to an Indigo Workstation

    Figure 2-14 Connecting the Multiplexer's SCSI Cable to an Indigo Workstation

    Figure 2-15. Connecting the Multiplexer's SCSI Cable to an Indigo2 Workstation

    Figure 2-15 Connecting the Multiplexer's SCSI Cable to an Indigo2 Workstation

  3. Attach the other end of the cable to the leftmost socket on the back of the multiplexer, as shown in Figure 2-13.

  4. If the customer has purchased two multiplexers, attach the cable of the second multiplexer to the middle socket on the first multiplexer.

  5. Cable the power supply to the rightmost port on the multiplexer, as shown in Figure 2-13.

  6. Select an appropriate SCSI ID number by inserting a small screwdriver or other tool and rotating the arrow, as shown in Figure 2-16. Using SCSI ID 7 on early models of the multiplexer causes the power indication LED to blink. Select another ID and the LED glows steadily with no blink.


    Note: The system disk in the Indigo workstation is always SCSI target ID 1 (on the Indy and Indigo2, ID 1 is the default). Never select SCSI ID 1 for a multiplexer.

    Figure 2-16. Setting the Multiplexer SCSI ID

    Figure 2-16 Setting the Multiplexer SCSI ID

  7. If the multiplexer is the last SCSI device on the workstation's standard external SCSI port, terminate it properly by attaching the external SCSI terminator included in the multiplexer shipment. See Figure 2-13.

  8. Reboot the workstation.

Cabling the IRISconsole Multiplexer and the Challenge Servers

Follow these steps:

  1. Attach a 25-pin end of a serial cable to serial port 1 (or another port, as the customer wishes) on the top of the multiplexer.

  2. Connect the 9-pin connector from serial port 1 on the multiplexer to the connector labeled System Console tty_1 on the I/O panel of the first OPS host (OPS1), as shown in Figure 2-17.

    Figure 2-17. Cabling the OPS Host Remote System Control Connector

    Figure 2-17 Cabling the OPS Host Remote System Control Connector

  3. Attach a 25-pin end of a serial cable to serial port 2 (or another port, as the customer wishes) on the top of the multiplexer. Connect the other end of this cable to the Remote Sys Control SSE Use Only connector, which is to the right of the I/O panel on the first OPS host, as shown in Figure 2-18.

    Figure 2-18. Cabling the OPS Host System Console Connector

    Figure 2-18 Cabling the OPS Host System Console Connector

    Figure 2-19 shows cabling for a Challenge XL.

    Figure 2-19. Multiplexer Cabling for Challenge XL

    Figure 2-19 Multiplexer Cabling for Challenge XL

  4. Repeat steps 1 through 3 for the second OPS host: attach serial cables from ports 3 and 4 on the multiplexer to the OPS2's System Console tty_1 and Remote Sys Control SSE Use Only ports, respectively.

    Figure 2-20 diagrams IRISconsole multiplexer cables.

    Figure 2-20. IRISconsole Cabling for OPS

    Figure 2-20 IRISconsole Cabling for OPS


    Note: Do not use the 8-pin powered serial connector in systems that have the IRISconsole multiplexer installed.


Powering On the Systems

To power on the component systems, follow these steps:

  1. Make sure that the Challenge RAID power switch is off.

  2. Connect the power cord to the storage system. Plug the other end of the power cord into an AC power outlet. See Figure 2-21.

    Figure 2-21. Connecting the Power Cord (Back of Challenge RAID Storage System)

    Figure 2-21 Connecting the Power Cord (Back of Challenge RAID Storage System)

  3. Turn on the storage system's power. The green power light on the front of the Challenge RAID storage system comes on, and the fans rotate. See Figure 2-22.

    Figure 2-22. Turning On the Storage System Power

    Figure 2-22 Turning On the Storage System Power

  4. If the busy light on none of the drive modules lights up, make sure that the power for each storage-control processor is enabled. Move the fan module's latch to the UNLOCK position, as indicated in Figure 2-23.

    Figure 2-23. Unlocking the Fan Module

    Figure 2-23 Unlocking the Fan Module

  5. Swing open the fan module and move the power switch for each disabled storage-control processor to the enable position, as shown in Figure 2-24.

    Figure 2-24. Enabling a Storage-Control Processor's Power

    Figure 2-24 Enabling a Storage-Control Processor's Power

  6. Swing the fan module closed, and then move the latch to the LOCK position.


    Caution: Do not leave the fan module open for more than two minutes at any time. The Challenge RAID storage system shuts down if the fan module is disabled for more than two minutes.


  7. If necessary, connect the power cables and power on the OPS hosts, the Challenge RAID, the workstation, and the multiplexer.

Testing the Installed OPS Hardware

To test the OPS installation, run hinv on a server. Figure 2-25 shows a sample output.

Figure 2-25. Example hinv Output for Installed OPS System

Figure 2-25 Example hinv Output for Installed OPS System