SGI systems that use the NUMAflex design (such as SGI Origin 3000 and SGI Onyx 3000 series systems) and run the IRIX 6.5 or later release can be partitioned. Partitioning divides a multi-node distributed shared memory (DSM) system into smaller systems without the need to physically recable the system. Each partition, which is logically separated from other partitions, runs its own operating system kernel and behaves as a standalone system. Each partition has its own console, root filesystem (system I- or IX-brick), and IP (internet protocol) address.
Partitioning provides the following benefits:
High availability. Any SGI NUMAflex server can be partitioned to appear as a cluster of systems. Unlike a cluster of systems, however, it benefits from the low-latency and high-bandwidth NUMAlink communication between partitions. NUMAlink and partitioning provide both high availability and high performance.
Job resilience. Jobs that run within a partition can use checkpointing to save job information, and each partition can be checkpointed individually as a single memory and application space. Checkpointed jobs can be restarted after the partition has been repaired, or they can be restarted in another partition. Load-balancing software (such as Load Sharing Facility [LSF] from Platform Computing) can manage workloads between partitions in a system the same as in a cluster. The IRIX Advanced Cluster Environment provides a checkpoint and restart mechanism for jobs that run on more than one partition.
No single point of failure. A single shared-memory system can be divided into separate partitions, eliminating the possibility of a single point of failure for the entire system.
Multiple environments. Because each partition in the system is running a separate operating system kernel, different IRIX kernels can be running in a single system at the same time, allowing the flexibility to manage all types of users and jobs. You can use one partition for development or experimental work and another partition for production work.
Fault isolation. Failures in one partition will not affect the other partitions. Upon an unrecoverable hardware- or software-related failure, the affected partition immediately shuts down, limiting the propagation of data corruption within the partition. Memory protection hardware within all partitions prevents bad packets from corrupting memory.
Figure 1-1 provides an example of a system that is divided into two partitions. Both partitions can be booted, powered off, powered on, and rebooted without affecting the normal operation of the other partition.
The NUMAlink interconnect is used for communication within partitions and between partitions. The NUMAlink interconnect uses memory-to-memory copy routines, so inter-partition communication is a high-performance alternative to I/O clustering. Each NUMAlink cable can transfer data at a rate of 3.2 GB/s.
For communication between partitions, you can use either of the following mechanisms:
Network interface driver using sockets (such as TCP/IP or UDP)
Shared-memory interface
Note: The TCP/IP interface supports both NFS and CXFS traffic between partitions.
Each partition must have the infrastructure to run as a standalone system. This infrastructure includes the following components:
At least one Cx-brick for system sizes of four Cx-bricks or less, or multiples of two Cx-bricks for system sizes greater than four Cx-bricks
Note: The Cx-bricks do not have to be fully populated with processors. One I- or IX-brick
One root disk that contains the operating system image
One console connection
You can configure your system as a single system, or you can divide it into partitions that operate as separate systems. For example, you can configure an Origin 3900 system that has 512 processors as a single 512-processor system, or you can divide it into as many as eight separate systems. The size of a partition can be configured to match your job size.
Reconfiguring partitions is performed through software only and does not require any cabling changes. For example, you can configure your system with multiple partitions during the day and then reconfigure the system as a single partition to run a large application at night. You can perform all of this configuring from a remote location. For more information about reconfiguring, see “Creating Partitions”.
The following configuration rules apply to partitions:
Each partition must have a unique ID number between 1 and 63, inclusively. An ID number of 0 indicates that the system is not partitioned.
The console of the lowest-numbered Cx-brick in a partition becomes the console for that partition. (The original console remains the console for the partition that contains it.)
Each partition has its own set of PROM environment variables: ConsolePath, OSLoadPartition, SystemPartition, netaddr, and root.
A partition must be contiguous in topology. The route between any two processors in the same partition must be contained within that partition, and not through any other partition. If the bricks in a partition are not contiguous, the system will not boot.
An I/O brick belongs to the partition to which the attached Cx-brick belongs. If an I/O brick is attached to two Cx-bricks, both Cx-bricks must be in the same partition. I/O bricks cannot be shared by two partitions.
Note: Peripherals, such as dual-ported disks, can be shared the same way that two nodes in a cluster can share peripherals. The G-brick must follow the same partitioning rules as the I/O bricks. Both graphics pipes on a G-brick must be connected to the same partition.
For Origin 3900 systems that contain more than 18 Cx-bricks, specific bricks must reside in each partition. In these configurations, the Cx-bricks reside in three or four compute racks. Partitions that consist of more than two Cx-bricks must be “striped” across the compute racks. For example, in a 512-processor system that is partitioned as four 128-processor partitions, the partitions consist of the following Cx-bricks (see Figure 1-2):
Table 1-1 lists the partition configurations that are available to the Origin 3900 system. The marketing codes for systems that contain eight Cx-bricks or less are also valid for Onyx 3900 graphics systems. Illustrations of the partitioned systems are provided in Chapter 2.
Table 1-1. Valid System Partitions
Marketing Code | Number of Cx-bricks in System | Number of Partitions | Number of Cx-bricks in Each Partition | Maximum Number of Processors in Partition | Illustrations |
|---|---|---|---|---|---|
Partition-O3900-C020 | 2 | 2 | 1 | 16 |
and Figure 2-2
|
Partition-O3900-D030 | 4 | 2 | 2 | 32 |
and Figure 2-4
|
Partition-O3900-D031 | 4 | 4 | 1 | 16 |
and Figure 2-10
|
Partition-O3900-D034 | 4 | 3 | 1 partition with 2 Cx-bricks 2 partitions with 1 Cx-brick | 32 16 |
and Figure 2-8
|
Partition-O3900-D037 | 4 | 2 | 1 partition with 3 Cx-bricks 1 partition with 1 Cx-brick | 48 16 |
and Figure 2-6
|
Partition-O3900-E050 | 8 | 2 | 4 | 64 |
and Figure 2-18
|
Partition-O3900-E051 | 8 | 4 | 2 | 32 |
and Figure 2-14
|
Partition-O3900-E054 | 8 | 3 | 1 partition with 4 Cx-bricks 2 partitions with 2 Cx-bricks | 64 32 |
and Figure 2-20
|
Partition-O3900-E057 | 8 | 2 | 1 partition with 6 Cx-bricks 1 partition with 2 Cx-bricks | 96 32 |
and Figure 2-16
|
Partition-O3900-F070 | 16 | 2 | 8 | 128 |
and Figure 2-24
|
Partition-O3900-F071 | 16 | 4 | 4 | 64 |
and Figure 2-26
|
Partition-O3900-F072 | 16 | 8 | 2 | 32 |
and Figure 2-30
|
Partition-O3900-F076 | 16 | 3 | 1 partition with 8 Cx-bricks 2 partitions with 4 Cx-bricks | 128 64 |
and Figure 2-32
|
Partition-O3900-F077 | 16 | 4 | 1 partition with 8 Cx-bricks 1 partition with 4 Cx-bricks 2 partitions with 2 Cx-bricks | 128 64 32 |
and Figure 2-34
|
Partition-O3900-F078 | 16 | 5 | 1 partition with 8 Cx-bricks 4 partitions with 2 Cx-bricks | 128 32 |
and Figure 2-36
|
Partition-O3900-F079 | 16 | 2 | 1 partition with 14 Cx-bricks 1 partition with 2 Cx-bricks | 224 32 |
and Figure 2-28
|
Partition-O3900-G090 | 32 | 2 | 16 | 256 |
and Figure 2-38
|
Partition-O3900-G091 | 32 | 4 | 8 | 128 |
and Figure 2-40
|
Partition-O3900-G092 | 32 | 8 | 4 | 64 |
and Figure 2-42
|
Partition-O3900-G095 | 32 | 3 | 1 partition with 16 Cx-bricks 2 partitions with 8 Cx-bricks | 256 128 |
and Figure 2-44
|
Partition-O3900-G096 | 32 | 4 | 1 partition with 16 Cx-bricks 1 partition with 8 Cx-bricks 2 partitions with 4 Cx-bricks | 256 128 64 |
and Figure 2-46
|
Partition-O3900-G097 | 32 | 5 | 1 partition with 16 Cx-bricks 1 partition with 8 Cx-bricks 1 partition with 4 Cx-bricks 2 partitions with 2 Cx-bricks | 256 128 64 32 |
and Figure 2-48
|
Partition-O3900-H105 | 6 | 3 | 2 | 32 |
and Figure 2-12
|
Partition-O3900-H120 | 12 | 5 | 1 partition with 4 Cx-bricks 4 partitions with 2 Cx-bricks | 64 32 |
and Figure 2-22
|
You can set up or reconfigure partitions by using the following commands:
part command from the L2 controller
mkpart partition configuration command from the IRIX operating system
setpart command from the command monitor
You can use the partmgr command to invoke a partitioning administration tool that enables you to define and store partition definitions.
| Note: The mkpart and partmgr commands can be run only by the system administrator with superuser privilege. If the system is configured as CAP_NO_SUPERUSER, you must have effective capabilities of CAP_SHUTDOWN, CAP_DEVICE_MGT, and CAP_SYSINFO_MGT. See the capabilities(4) man page for more information on the capability mechanism that provides fine-grained control over the privileges of a process. |
Each partition has its own system console from which you can run these commands, or you can use SGIconsole to manage multiple partitions.
To partition a system via an L2 controller, use the part command to set the partition ID for each Cx-brick in a partition. For example, to place the Cx-brick in rack 001, slot U07, in partition 1, enter the following command:
L2>r 1 s 7 brick part 1
| Note: To access the L2 controller, type Ctrl t. The L2 prompt (L2>) should appear on the screen. |
The following list of commands partitions a 512-processor system into four 128-processor partitions:
L2>r 1 s 7,11 brick part 1
L2>r 2 s 7,11 brick part 1
L2>r 3 s 7,11 brick part 1
L2>r 4 s 7,11 brick part 1
L2>r 1 s 15,19 brick part 2
L2>r 2 s 15,19 brick part 2
L2>r 3 s 15,19 brick part 2
L2>r 4 s 15,19 brick part 2
L2>r 1 s 23,27 brick part 3
L2>r 2 s 23,27 brick part 3
L2>r 3 s 23,27 brick part 3
L2>r 4 s 23,27 brick part 3
L2>r 1 s 31,35 brick part 4
L2>r 2 s 31,35 brick part 4
L2>r 3 s 31,35 brick part 4
L2>r 4 s 31,35 brick part 4
You can verify that the partition IDs are set correctly by using the cfg L2 controller command. This command identifies partitions by placing a number after the brick identifier. For example, 001c07.1 indicates that the Cx-brick in slot 07 of rack 001 is in partition 1, and 001c35.4 indicates that the Cx-brick in slot 35 of rack 001 is in partition 4.
In addition, you can use the sel L2 command to list the system consoles in a partitioned system. For example, the following output indicates the system consoles for a system that contains two partitions:
L2>sel known system consoles (partitioned) Partition 1: 001c07 - L2 detected Partition 2: 001c11 - L2 detected current system console console input: partition 2, 001c11 console0 console output: any brick partition 2 L2> |
To enable communication between a partition and the L2 controller, use the sel p L2 command. This command selects which partition is the console. For example, enter sel p 1 to select partition 1 as the console. When partition 1 is the console, you can access any brick within this partition via the L2 controller.
| Note: To select each partition as the console, you must use multiple windows, one window for each partition. Because each partition acts as a standalone system, these windows enable you to access each partition via the L2 controller. |
The mkpart partition configuration command allows you to perform the following tasks:
List the current partitions.
Save the current configuration.
Set up partitions as follows:
Split an existing system into multiple partitions.
Coalesce partitions into one partition.
Split up existing partitions and regroup them differently.
See the mkpart(1M) man page for detailed information on this command. Table 1-2 lists the mkpart command options.
Table 1-2. mkpart Command Options
Option | Description |
|---|---|
-p partid | Specifies the partition ID for the new configuration. |
-m brick | Indicates the brick number. |
-i | Coalesces all currently active partitions into a single system with a partition ID of 0. A partition ID of 0 indicates a system that is not partitioned. |
-l | Lists current partitions. |
-n | Prints partition ID of this partition. |
-h | Prints a usage message. |
-F | Forces override of all sanity checks. The command sets up partitioning as given and tries to reboot the system. Caution: Use the -F option only if you fully understand the consequences of using this option incorrectly. |
A typical sequence of partitioning commands is as follows:
Run mkpart -l to list all partitions.
Run mkpart again to repartition the system.
Partition IDs are stored in the L1 controllers.
mkpart uses a daemon (mkpd) to write IDs on remote partition L1 controllers.
Reboot all affected partitions if mkpart is successful.
During the boot operation, each Cx-brick in a system is assigned a module identifier (ID) and a partition ID that is maintained by the brick's L1 controller. By design, the module ID is unique for each Cx-brick, while all Cx-bricks in a single partition have the same partition ID.
When a partition is booted initially, a dynamic discovery routine identifies new hardware in the system. After the discovery process, a reset fence is created between a hub or router on the other side of the link that has a different partition ID. The dynamic discovery and fence generation for a partition occurs while the other partitions continue operation.
You can also set up partitions by using the setpart command from the command monitor. To use the setpart command, follow these steps:
Reset the system.
From the System Maintenance menu, enter option 5 for the command monitor.
Run the setpart command. (The setpart -h command displays all setpart options.) The setpart command is an interactive command that receives your input, including a partition number and the module identifier of the Cx-bricks that make up the partition.
Reset the system again and the system will function as two or more partitions.
To coalesce multiple partitions into one partition, follow these steps:
Access the command monitor for all the partitions.
Run the setpart -c command in any of the partitions. This command clears all partition information and offers to reset all partitions simultaneously. After the reset, the system is configured as a single partition.
The partmgr command invokes a partitioning administration tool for the Origin 3000 series of servers that enables you to define and store partition definitions, including the following:
Partition ID information for bricks and racks.
Associated PROM environment variables.
After using the partmgr command, you must reboot all the affected partitions in order to change the system to the new partition scheme.
| Note: Only a system administrator with superuser privilege can use this command. If the system is configured as CAP_NO_SUPERUSER, you must have effective capabilities of CAP_SHUTDOWN, CAP_DEVICE_MGT, and CAP_SYSINFO_MGT. See the capabilities(4) man page for more information on the capability mechanism that provides fine-grained control over the privileges of a process. |
 | Caution: If you use the partmgr command incorrectly, this command can completely freeze all partitions in the system and prevent the partitions from booting the PROM. |
To use the partmgr command, you must be sure that the following requirements are met:
You must install the eoe.sw.partition software package, after which you must configure a new kernel and boot the system.
The L1 and L2 firmware must be current.
You must enable partitioning administration in all L1 controllers using the partdb enable L1 command.
The mkpd daemon must be running on all partitions for partmgr to work. If it is not running, the partmgr command may not be able to shut down remote partitions.
The PROM environment variable AutoPROMVars must be set to 1. This enables the PROM to use the environment variables defined in the mode definition.
See the partmgr(1M) man page for detailed information on this command. Table 1-3 describes the command options.
Table 1-3. partmgr Command Options
Option | Description |
|---|---|
-h | Help. Prints a usage message. |
-l | Lists brick IDs of all known system bricks to standard out. |
-a | Lists all currently defined modes. |
-n | Displays the current partition ID. |
-v | Displays the environment variables for the current partition. |
-c | Displays mode configuration. If no mode is specified (-m modename), the current mode is displayed. |
-x modename | Deletes a mode. |
-s modename | Sets a mode as the current mode. This may require some partitions to be rebooted. |
-i | Same as “-s Unpartitioned.” Initializes the system to the default unpartitioned state. |
-d | Defines a new node. |
-m modename | Specifies a mode name. This option is used with the -c and -d options. |
-u | When you define a new mode, this option specifies an unpartitioned system. |
-p part | Specifies a partition ID. A partition ID is a number between 1 and 63, inclusive. The order of the -p, -b, -r, and -e options is significant. See the partmgr(1M) man page. |
-b bricks | Specifies brick IDs (example: 001c13). |
-r racks | Specifies rack IDs (example: 001). |
-e var=value | Specifies an environment variable name/value pair. |
-f file | When you define a new mode, this option specifies an input file. |
-F | Forces override of some sanity checks. Use this option with caution and only if you fully understand the consequences of using this option incorrectly. |