An Array comprises the following hardware:

A complete Array system is shown schematically in Figure 1-1.

The Array 3.0 software binds the Array system hardware into a supercomputer that can be programmed and administered as one system. An Array system using Array 3.0 software is based on the following major components:

Many optional software packages are available from Silicon Graphics, Inc. to extend Array 3.0, including:

A variety of software packages from third parties also are available, including:

Most of these components are described at more length in following topics.

The basic computational building block of an Array is a Silicon Graphics, Inc. multiprocessor. Any system running IRIX 6.2 can participate as a node in Array 3.0, but normally a node is a multiprocessor system. Depending on the type of Array and customer's choices, a node can be any of the systems listed in Table 1-1.

Table 1-2 lists information sources for the different types of systems.

Array nodes are normally connected by a high-performance, dual-channel HIPPI network. Each node is equipped with one or more bidirectional HIPPI interfaces. Each interface provides 100 MB per second of data bandwidth in either direction.

The HIPPI interfaces are connected via a high-performance HIPPI crossbar switch (optional in a two-node Array). The HIPPI switch is nonblocking, with sub-microsecond connection delays. The network appears to be fully connected and contention occurs only when two sources send data to the same destination at the same time.

IRIX 6.2 or IRIX 6.4 and the Array 3.0 software provide protocol layers and APIs to access the HIPPI network, including direct physical layer, HIPPI framing protocol, and TCP/IP. The HIPPI support includes special bypass capabilities to expedite transmission of short messages. The bypass capability is transparent to the applications using it.

Table 1-3 lists information sources on HIPPI and the HIPPI crossbar (which is produced by a third party, Essential Communications, Inc.).

Array nodes can be configured with hardware graphics support, to provide two and three-dimensional visualization performance commensurate with the available compute power. The available graphics options are listed in Table 1-1. Complex supercomputing visualization architectures can be built by aggregating compute and graphics nodes, as illustrated in Figure 1-2.

An IRISconsole serves as a single, centralized administrative console for Array administration and maintenance. The IRISconsole consists of an O2 or Indy workstation, an IRISconsole multiplexer box, and the IRISconsole graphical cluster management software. From the IRISconsole, administrators can control, configure, monitor, and maintain the individual Array nodes.

The O2 or Indy workstation serves as a virtual console for each node. The workstation is connected to the multiplexer via a SCSI interface. The multiplexer in turn connects to the Remote System Control port of each node. Commands from the console workstation are routed to the appropriate node, and results from the nodes are routed back.

The IRISconsole graphical user interface provides a convenient graphic representation of the array. Sets of nodes can be selected and operated upon. You can open a command window directly to any node. You can use the IRISconsole graphical interface to

For more about the features of IRISconsole and illustrations of its use, see “Using the IRISconsole Workstation”. Table 1-4 lists other information sources for IRISconsole and its hardware.

Array Services includes administrator commands, libraries, daemons and kernel extensions that support the execution of programs across an Array.

A central concept in Array Services is the array session handle (ASH), a number that is used to logically group related processes that may be distributed across multiple systems. The ASH creates a global process namespace across the Array, facilitating accounting and administration.

Array Services also provides an array configuration database, listing the nodes comprising an array. Array inventory inquiry functions provide a centralized, canonical view of the configuration of each node. Other array utilities let the administrator query and manipulate distributed array applications.

The Array Services package comprises the following primary components:

The use of the ainfo, array, arshell, and aview commands is covered in Chapter 2, “Using an Array.” The use of the libarray library is covered in Chapter 4, “Performance-Driven Programming in Array 3.0.”

Performance Co-Pilot (PCP) is a Silicon Graphics product for monitoring, visualizing, and managing systems performance.

PCP has a distributed client-server architecture, with performance data collected from a set of servers and displayed on visualization clients. Performance data can be obtained from multiple sources, including the IRIX kernel and user applications. With support for low-intrusion performance data collection, reduction, and analysis, PCP permits a variety of metrics to be captured, correlated, reduced, recorded, and rendered.

PCP has been customized for Array systems to provide visualization of system-level and job-level statistics across the array. An array user can view a variety of relevant performance metrics on the array via the following utilities:

For more information about Performance Copilot, see The Performance Co-Pilot User's and Administrator's Guide (007-2614-xxx).

SHARE II, a “Fair Share” scheduler, allows an organization to create its own resource allocation policy based on its assessment of how resource usage should be fairly distributed to individuals or arbitrarily grouped users. SHARE II is available only for Arrays that use IRIX 6.2; it is not available for IRIX 6.4.

With SHARE II, users are grouped into a system-wide resource allocation and charging hierarchy. The hierarchy can represent projects, divisions, or arbitrary sets of users. Within this hierarchy, resource usage policy can be varied or delegated at any level according to organizational priorities.

Users can be limited in consumption of renewable resources (such as printer pages) and fixed resources (such as instantaneous memory use). Other limits are imposed during periods of scarcity (for example, CPU run time during periods of contention). Thus, SHARE II provides a fair share of the system resources during high-load periods without overcommitment, wasteful static reservations, or expensive administrator intervention.

PerfAcct, a third-party software product, gathers system accounting data from all nodes in a central location, where it is summarized and used to generate usage reports or billing. PerfAcct exploits IRIX extended-session accounting data to provide true job accounting. Job and project accounting permits usage tracking and billing by external or internal contracts, departments, tasks, and projects.

PerfAcct features low-overhead data collection on the nodes being monitored. To minimize system load on the monitored systems, archiving and summarization can be put on a remote low-cost workstation. PerfAcct also includes aggregate accounting statistics, as well as graphical user interface tools for measuring dynamic system load.

Table 1-5 lists information sources for the management tool products.

Users can log in to a node to execute jobs interactively using normal IRIX job-control facilities. Interactive jobs can be command-line based, or can be X Windows applications that execute on the node but display on the user's workstation.

Jobs started interactively can be sequential programs, or multi-threaded programs executing within a node, or distributed-memory parallel applications executing across several nodes. Distributed programs using MPI or PVM can be started and monitored using the graphical monitors XMPI and XPVM; these display job status graphically on the user's workstation screen. Table 1-6 lists information sources on interactive processing.

Batch processing allows off-line job scheduling. Batch processing is appropriate for production environments, high job-load environments, and situations where program results are not required immediately.

When an Array system is used for batch scheduling, users submit jobs to batch queues, which contain ordered sets of waiting jobs. When sufficient compute resources become available, and subject to tunable scheduling constraints, jobs are extracted from the batch queues and scheduled on the nodes. Job results and termination status are recorded in files or are electronically mailed to the user. See Figure 1-3.

Several popular batch facilities are compatible with Array 3.0, including the Network Queuing Environment (NQE) from Silicon Graphics, Inc.; the Codine^™ Job-Management System from Genias Software, Inc.; and Load Sharing Facility (LSF^™) from Platform Computing, Inc.

NQE consists of the following components that provide a seamless environment for users of the Array:

IRIX Checkpoint and Restart (CPR) facility allows you to save the status of long-running jobs and restart them easily.

Table 1-7 lists information sources on these products.

The MIPSpro^™ compilers are the third-generation family of optimizing and parallelizing compilers from Silicon Graphics, offering comprehensive support for parallel application development.

Exploiting aggressive dependency analysis, the compilers perform automatic program restructuring, software pipelining, and parallelization. The compilers also provide a comprehensive set of comment directives that enable users to assist the compiler in the parallelization process.

Silicon Graphics, Inc. offers MIPSpro compilers for Fortran 77, Fortran 90, and C; as well as compilers for Ada 95, C++, assembly language, and Pascal. For detailed information about each compiler see the sources listed in Table 1-8.

High Performance Fortran (HPF) is an extended version of Fortran 90 that is emerging as a standard for programming of shared- and distributed-memory systems in the data-parallel style. HPF incorporates a data-mapping model and associated directives that allow a programmer to specify how data is logically distributed in an application. An HPF compiler interprets these directives to generate code that minimizes interprocessor communication in distributed systems and maximizes data reuse in all types of systems.

HPF compilers are available for Array systems from the Portland Group, Inc. and Applied Parallel Research, Inc. Table 1-9 lists information sources for these products.

The compilers are complemented by CHALLENGEcomplib^™, a comprehensive, optimized collection of scientific and math subroutine libraries popular in scientific computing. The library consists of two subcomponents: SGIMATH and SLATEC.

SGIMATH is hand-tuned, optimized, and parallelized, providing high-performance, portable implementations of the following popular numerical facilities:

A source for a more detailed overview of CHALLENGEcomplib is listed in Table 1-10. Most of the functions within the library are documented in reference pages that install with the product.

The primary process control services of the Array are provided by the IRIX operating system, a symmetric multiprocessing operating system based on UNIX SVR4 with compatibility for BSD.

IRIX version 6.2 is required for Array 3.0 on Challenge/Onyx systems, and IRIX 6.4 on Origin systems. This version provides fast, flexible support for shared-memory interprocess communication, high-performance I/O, and performance-centric scheduling. Within a node, related processes are gang-scheduled to prevent one process from wasting time by spinning on locks held by blocked peers. Process placement decisions incorporate cache affinity heuristics, which minimizes multiprogramming-induced cache thrashing by tending to keep particular processes on the same processor.

Real-time processing can be supported with the REACT facilities, including nondegrading priorities, deadline scheduling, and reliably bounded kernel latencies. Hooks to support optional SHARE II Fair Share Scheduling checkpoint-restart facilities are also supported.

IRIX supports a variety of system functions to allow shared memory interprocess communication (IPC) between processes within one node. SVR4-compatible library functions for semaphores, message queues, and shared memory are supported. High-performance IRIX-unique facilities for shared memory, semaphores, and mutex locks are included. POSIX-compatible library functions for semaphores, message queues, and shared memory are integrated into IRIX 6.4 (available as a patch set for IRIX 6.2).

Overview sources on IRIX and on the REACT real-time programming extensions are listed in Table 1-11.

Silicon Graphics includes a powerful set of parallel debugging, profiling, and visualization tools as part of the Developer Magic^™ application development suite. Systemic performance visualization is provided by the Performance Co-Pilot facility, and array extensions.

In addition to these, IRIX 6.2 contains the interactive debugger dbx and profiling tools pixie and prof. Information sources on developer tools are listed in Table 1-12.