The first step in tuning a parallel application—and it is a large step—is to tune it for best performance on a single node. Tuning any program begins with instrumenting the program to identify the parts where it spends excess time. Analyze these parts to determine whether algorithmic changes are possible. Optimizations of program logic and algorithms, when they are possible, always yield the largest improvements at the lowest cost.

When you are sure the algorithm is optimal and its coding is logically correct, examine the program for library use, cache use, software pipelining, and SMP performance.

When a program runs efficiently on a single node, but you find you still need to recruit more CPU cycles to it, you can look for further performance gains through multinode parallelism.

Be aware first that multiparallelism is far from a panacea. It is important to consider Amdahl's Law. If the code that can be run in parallel consumes less than 95% of the total execution time, targeting fewer than 18 processors is sufficient to realize any potential benefit from parallelization. When this is the case, parallelism within a single node is the most appropriate strategy (presuming at least one node has sufficient CPUs installed).

When the parallelization potential is above 96%, then parallelizing across the nodes of the Array can result in a performance speedup.

When designing a multinode application, your basic strategy should be to maximize the work done on the data within any node before communication between nodes is required. In general, aim to communicate between nodes as rarely as possible, and when communication is needed, to use the largest message units possible. This strategy maximizes the use of the high bandwidth and lower latencies of the bus within a node, as compared to the relatively slower communications of the HIPPI network.

While the implementation of your internode parallel algorithms may use a shared memory model (HPF) or message-passing model (MPI, PVM), the design of the fundamental algorithms is conceptually similar to the design of good parallel algorithms for a single-node program. To achieve the performance advantages of distributed multiprocessing, you must

If you design your applications to accept variable numbers of processors and problem sizes, often you can debug your fundamental program logic within a single node. When executing within one node you can apply all the debugging and visualization tools of the Workshop suite.

Several models of parallel computation are available for the IRIX and Array 3.0. These models are discussed and compared in detail in the book listed in Table 4-5.

The models are summarized here, but read the chapter of the book shown for details and the latest version information.

IRIX provides several facilities that permit parallel processes to communicate by directly reading and writing the same address space.

The message-passing communication model provides a “mail delivery” paradigm for interprocess communication. A collection of data items is given an identifier and “mailed” to a destination process, which subsequently receives it.

The message-passing model makes a clean separation between program modules, affording some protection from accidental changes to shared memory. However, any message-passing facility must incur some delay compared to shared memory, due to buffering, abstraction, and (sometimes) copying and transmission overheads.

If you are preparing a new application, Silicon Graphics, Inc. recommends that you plan as follows:

There is no requirement that applications use any one set of facilities exclusively. For example, the following common models are possible, among others:

However, when designing a program to use a hybrid model, you must be aware that the MPI library is not “thread-safe,” that is, it has global variables with values that can be destroyed if it is executed by two lightweight processes concurrently. The MPI library should be entered by only one process in any share group. This is discussed in more detail in the MPI and PVM User's Guide, 007-3286-xxx.

Array 3.0 contains an optimized protocol stack supporting MPI protocols on HIPPI. Separate design approaches have been implemented for short messages and long ones. Special attention is paid to latency for shorter messages, which are more common. The advice given in “Reducing the Effect of Communication Delay”, to use fewer, longer messages, is valid, but the reduced latency from prior versions should improve the performance of many programs.

Other conditions can affect the use of HIPPI by MPI. When four or more applications are contending for the use of an adapter, MPI does not use that adapter. (The limit is 8 applications per adapter on an Origin2000 node). When a node does not have a HIPPI adapter, or when the maximum MPI jobs are contending for all available adapters, an internode MPI transfer uses a socket instead. This case is not optimized and will be slower.

The basic IRIX support for TCP/IP is not changed for Array 3.0, and does not take advantage of the special MPI tuning. As a result, you are strongly advised to construct a distributed application using MPI, not sockets.

When you must use sockets, be aware that the bandwidth you can achieve can vary over an extremely wide range depending on several factors. The most important are: the size of the socket buffer (the SO_SNDBUF and SO_RCVBUF options of setsockopt()) and the size of the message.

Typical performance with a 62 KB socket buffer and a stream of 16 KB messages is approximately 15 MB/second. Much higher speeds, up to 60 MB/sec and more, can be achieved using larger, page-aligned transfers, with buffer pages locked in memory. Rates of 90 MB/sec can be achieved by highly-tuned benchmark programs.

To effectively exploit the memory hierarchy of the Array, you must be aware of the effects of communication latency and bandwidth while designing your program. Your best strategy is to send fewer, longer messages, and to overlap communication with useful work when possible.

Overlap Processing with Communication

Try to design the application to overlap message delays with computation. MPI permits a process to send messages asynchronously. The MPI_Isend() function returns from the immediately, before the message has been sent. MPI also permits asynchronous receipt; the MPI_Irecv() function tests for available data without waiting when none is ready.

Asynchronous communication permits the sender or receiver to continue computation while data is transferred by the system. Figure 4-1 shows the potential performance advantages of asynchronous communication. Figure 4-1a is a time line of two standard communicating processes. Figure 4-1b is a time line of the same two processes using asynchronous sends and receives. Here, much of the communication time is hidden behind computation.

Figure 4-1. Gaining Efficiency Through Asynchronous Communication

When implementing asynchronous communication in MPI, it i s important to use both MPI_Isend() and MPI_Irecv(). Neither MPI_Send() nor MPI_Isend can begin to transfer data until a matching receive has been posted. Using MPI_Irecv() allows your program to post a receive earlier rather than later.

Asynchronous communication can in practice hide most communication delays, but it can require program restructuring. Optimize an existing program in other ways first; possibly the increased complexity in program logic will not be necessary.

The programming interface to Array Services is declared in the header file /usr/include/arraysvcs.h. The object code is located in /usr/lib/libarray.so, included in a program by specifying -larray during compilation. The library is distributed in o32, n32, and 64-bit versions (not all need to be installed). The functions are documented in reference pages in volume 3.

The library functions can be grouped into these categories:

These functions are examined in following topics.

The functions listed in Table 4-8 are used to open a connection between the node where your program runs and an instance of arrayd in the same or another node.

The key function is asopenserver(). It takes a nodename as a character string (as a user would give it in the -s option; see “Summary of Common Command Options”), and optionally a socket number to override the default arrayd socket number. This function opens a socket connection to the specified instance of arrayd. The returned token (type asserver_t) stands for that connection and is passed to other functions.

The functions for getting and setting server options can change the configured options shown in Table 4-9. To set these options is the programmatic equivalent of passing command line options in an Array Services command (see “About Array Configuration” and “Summary of Common Command Options”).

The functions summarized in Table 4-10 are used to interrogate the configuration database used by arrayd in a specified node (see “About Array Configuration”).

Using these functions you can extract any arrayname, nodename, or attribute that is known to an arrayd instance you have opened.

The functions summarized in Table 4-11 are used to create and interrogate ASH values.

The asallocash() function is like the command ainfo newash (see “About Array Session Handles (ASH)”). Only a program with root privilege can use the setash() system function to change the ASH of the current process. Unprivileged processes can create new ASH values but cannot change their ASH.

The functions summarized in Table 4-12 are used to enumerate the active ASH values at a specified node. In each case, the list of ASH values is returned in an asashlist_t structure.

The ascommand() function is the programmatic equivalent of the array command (see “Operation of Array Commands” and the array(1) reference page). This command has many options and can be used to execute commands in three distinct modes.

The command to be executed must be prepared in an ascmdreq_t structure, which contains the following fields:

typedef struct ascmdreq {
   char *array;            /* Name of target array */
   int flags;              /* Option flags */
   int numargs;            /* Number of arguments */
   char **args;            /* Cmd arguments (ala argv) */
   int ioflags;            /* I/O flags for interactive commands */
   char rsrvd[100];        /* reserved for expansion: init to 0's */
} ascmdreq_t;

Your program must prepare this structure in order to execute a command. The option flags allow for the same controls as the command line options of array.

The result of the command is returned as an ascmdrsltlist_t structure, which is a vector of ascmdrslt_t structures, one for each node at which the command was executed. Each ascmdrslt_t contains the following fields:

typedef struct ascmdrslt {
    char     *machine;  /* Name of responding machine */
    ash_t    ash;       /* ASH of running command */
    int      flags;     /* Result flags */
    aserror_t error;    /* Error code for this command */
    int      status;    /* Exit status */
    char     *outfile;  /* Name of output file */
    int      ioflags;   /* I/O connections (see ascmdreq_t) */
    int      stdinfd;   /* File descriptor for command's stdin */
    int      stdoutfd;  /* File descriptor for command's stdout */
    int      stderrfd;  /* File descriptor for command's stderr */
    int      signalfd;  /* File descriptor for sending signals */
} ascmdrslt_t;

The fields machine, ash, flags, error, and status reflect the result of the command execution in that machine. The other fields depend on the mode of execution.

The asrcmd() function allows a program to initiate any user command string on a specified node. This provides a powerful facility for remote execution that does not require root privilege, as the standard rcmd() function does (compare the asrcmd(3) and rcmd(3) reference pages).

The asrcmd() function takes arguments specifying:

The returned value (as with rcmd()) is a socket that represents the standard input and output streams of the executing command. Optionally, a separate socket for the standard error stream can be obtained.