In order to use an Array, you need the following items of information:

You can learn the hostnames of the array nodes if you know the array name, using the ainfo command:

ainfo -a arrayname machines

Each node in an Array is a Silicon Graphics, Inc. multiprocessor system such as an Origin2000. Each node has an associated hostname and IP network address. Typically, you use an Array by logging in to one node directly, or by logging in remotely from another host (such as the Array console or a networked workstation). For example, from a workstation on the same network, this command would log you in to the node named hydra6:

rlogin hydra6

For details of the rlogin command, see the reference page rlogin(1).

The system administrators of your Array may choose to disallow direct node logins in order to schedule array resources. If your site is configured to disallow direct node logins, your administrators will be able to tell you how you are expected to submit work to the array—perhaps through remote execution software or batch queueing facilities.

Once you have access to an Array you can invoke programs of several classes:

If you are allowed to do so, you can invoke programs explicitly from a logged-in shell command line; or you may use remote execution or a batch queueing system.

Programs that are X-windows clients must be started from an X server, either an X-terminal or a workstation running X Windows.

Some application classes may require input in the form of command line options, environment variables, or support files upon execution. For example:

Some information sources on program invocation are listed in Table 2-1.

You query the status of processes using the IRIX command ps. To generate a full list of all processes on a local system, use a command such as

ps -elfj

You can monitor the activity of processes using the command top (an ASCII display in a terminal window) or gr_top (displays in a graphical window).

For a global picture of the state of one node you can use gr_osview. It displays a variety of resource use values as histograms or bar-graphs in a graphical window. The command gmemusage displays memory use by all applications in the node where you start it.

You can start a process at a reduced priority, so that it interferes less with other processes, using the nice command. If you use the csh shell, specify /usr/bin/nice to avoid the built-in shell command nice. To start a whole shell at low priority, use a command like

/usr/bin/nice /bin/sh

You can schedule commands to run at specific times using the at command. You can kill or stop processes using the kill command. To destroy the process with PID 13032, use a command such as

kill -KILL 13032

Table 2-2 summarizes information about local process management.

NQE is usually installed in /usr/craysoft/nqe/bin on IRIX workstations. If that directory doesn't exist, contact your system administrator to see if NQE is installed and where it is installed. If /usr/craysoft/nqe/bin does exit, add it to your PATH variable. For example:

% setenv PATH $PATH:/usr/craysoft/nqe/bin 

or

$ export PATH=$PATH:/usr/craysoft/nqe/bin

The easiest way to start using NQE is through its graphical interface as implemented by the nqe command (see the nqe(1) reference page). If you run nqe on your workstation, you just start it. If you need to start nqe on an array node, with output to your workstation, you may need to set your DISPLAY variable first, as shown in the following example:

% setenv DISPLAY myworkstation:0
% nqe

Figure 2-1 shows the initial (top-level) NQE button bar window that should immediately appear.

Figure 2-1. NQE Top-level Window (Button Bar)

To see the status of jobs running under NQE, click on the Status button. Figure 2-2 shows an example of the Status window.

Figure 2-2. NQE Status Window

The example Status Window displays two jobs. Both are executing on the server homegrown and both are running (or will run) under the user account guest.

To refresh the status display, use the Refresh button in the Status window. You may also have the display refreshed periodically by setting the refresh option in the NQE Configuration Information Window, shown in Figure 2-3. Access the NQE Configuration Information Window using the Config button on the NQE button bar.

Figure 2-3. NQE Configuration Information Window

The slide bar titled “Status Refresh Rate” (in Figure 2-3) sets the refresh rate to a value other than 0. If the rate is set to 60, the NQE status display will be refreshed every 60 seconds.

To submit a new batch job, display the Submit window (accessed using the Submit button in the NQE button bar). Figure 2-4 shows an example of the Submit window with a sample job script. To submit the job, click the Submit button.

Figure 2-4. NQE Submit Window

A few details of the example job script shown in Figure 2-4 are of interest. The #QSUB string is an NQE directive, used to embed command line options within the script. (See the cqsub(1) or qsub(1) reference page for more information on embedded options.) The line

 #QSUB -a 8:05 

indicates to NQE that the job request should be started sometime after 8pm (20:00). The line

#QSUB -A nqearray

indicates to NQE that the job should run using the project “nqearray”. (See the projects(5) reference page for more information on project names.)

You can also operate NQE using a command-line interface. The NQE commands are summarized in Table 2-3. For details of the command-line interface, see the NQE User's Guide.

As noted under “Distributed Management Tools”, Array Services is composed of a daemon—a background process that is started at boot time in every node—and a set of commands such as ainfo. The commands call on the daemon process in each node to get the information they need.

One concept that is basic to Array Services is the array session, which is a term for all the processes of one application, wherever they may execute. Normally, your login shell, with the programs you start from it, constitutes an array session. A batch job is an array session; and you can create a new shell with a new array session identity.

Each session is identified by an array session handle (ASH), a number that identifies any process that is part of that session. You use the ASH to query and to control all the processes of a program, even when they are running in different nodes.

Each node is an IRIX server, and as such has a hostname. The hostname of a node is returned by the hostname command executed in that node:

% hostname
tokyo

The command is simple and documented in the hostname(1) reference page. The more complicated issues of hostname syntax, and of how hostnames are resolved to hardware addresses are covered in hostname(5).

An Array system as a whole has a name too. In most installations there is only a single Array, and you never need to specify which Array you mean. However, it is possible to have multiple Arrays available on a network, and you can direct Array Services commands to a specific Array.

It is possible for the Array administrator to establish an authentication code, which is a 64-bit number, for all or some of the nodes in an array (see “Configuring Authentication Codes”). When this is done, each use of an Array Services command must specify the appropriate authentication key, as a command option, for the nodes it uses. Your system administrator will tell you if this is necessary.

The commands of Array Services—ainfo, array, arshell, aview, and newsess—have a consistent set of command options. Table 2-5 is a summary of these options. Not all options are valid with all commands; and each command has unique options besides those shown. The default values of some options are set by environment variables listed in the next topic.

The Array Services commands depend on environment variables to define default values for the less-common command options. These variables are summarized in Table 2-6.

If your network includes more than one Array system, you can use ainfo arrays at one array node to list all the Array names that are configured, as in the following example.

homegrown% ainfo arrays
Arrays known to array services daemon
ARRAY DevArray
    IDENT 0x3381
ARRAY BigDevArray
    IDENT 0x7456
ARRAY test
    IDENT 0x655e

Array names are configured into the array database by the administrator. Different Arrays might know different sets of other Array names.

You can use ainfo machines to learn the names and some features of all nodes in the current Array, as in the following example.

homegrown 175% ainfo -b machines 
machine homegrown homegrown 5434 192.48.165.36 0
machine disarray disarray 5434 192.48.165.62 0
machine datarray datarray 5434 192.48.165.64 0
machine tokyo tokyo 5434 150.166.39.39 0

In this example, the -b option of ainfo is used to get a concise display.

You can use ainfo nodeinfo to request detailed information about one or all nodes in the array. To get information about the local node, use ainfo -l nodeinfo. However, to get information about only a particular other node, for example node tokyo, use -l and -s, as in the following example. (The example has been edited for brevity.)

homegrown 181% ainfo -s tokyo -l nodeinfo
Node information for server on machine "tokyo"
MACHINE tokyo
    VERSION  1.2
    8 PROCESSOR BOARDS
        BOARD: TYPE 15   SPEED 190
            CPU:   TYPE 9   REVISION 2.4
            FPU:   TYPE 9   REVISION 0.0
...
    16 IP INTERFACES  HOSTNAME tokyo   HOSTID 0xc01a5035
        DEVICE  et0    NETWORK    150.166.39.0    ADDRESS   150.166.39.39  UP
        DEVICE atm0    NETWORK 255.255.255.255    ADDRESS         0.0.0.0  UP
        DEVICE atm1    NETWORK 255.255.255.255    ADDRESS         0.0.0.0  UP
...
    0 GRAPHICS INTERFACES
    MEMORY
        512 MB MAIN MEMORY
        INTERLEAVE 4

If the -l option is omitted, the destination node will return information about every node that it knows.

The IRIX commands who, top, and uptime are commonly used to get information about users and workload on one server. The array command offers Array-wide equivalents to these commands.

homegrown 180% array -s tokyo -l who 
joecd    tokyo        frummage.eng.sgi -tcsh 
joecd    tokyo        frummage.eng.sgi -tcsh 
benf     tokyo        einstein.ued.sgi. /bin/tcsh 
yohn     tokyo        rayleigh.eng.sg vi +153 fs/procfs/prd
...

homegrown 181% array uptime 
   homegrown:  up 1 day,  7:40,  26 users,  load average: 7.21, 6.35, 4.72
    disarray:  up  2:53,  0 user,  load average: 0.00, 0.00, 0.00
    datarray:  up  5:34,  1 user,  load average: 0.00, 0.00, 0.00
       tokyo:  up 7 days,  9:11,  17 users,  load average: 0.15, 0.31, 0.29
homegrown 182% array -l -s tokyo uptime 
       tokyo:  up 7 days,  9:11,  17 users,  load average: 0.12, 0.30, 0.28

homegrown 183% array top 
        ASH        Host           PID User       %CPU Command
----------------------------------------------------------------
0x1111ffff00000000 homegrown        5 root       1.20 vfs_sync
0x1111ffff000001e9 homegrown     1327 guest      1.19 atop
0x1111ffff000001e9 tokyo        19816 guest      0.73 atop
0x1111ffff000001e9 disarray      1106 guest      0.47 atop
0x1111ffff000001e9 datarray      1423 guest      0.42 atop
0x1111ffff00000000 homegrown       20 root       0.41 ShareII
0x1111ffff000000c0 homegrown    29683 kchang     0.37 ld
0x1111ffff0000001e homegrown     1324 root       0.17 arrayd
0x1111ffff00000000 homegrown      229 root       0.14 routed
0x1111ffff00000000 homegrown       19 root       0.09 pdflush
0x1111ffff000001e9 disarray      1105 guest      0.02 atopm

The ArrayView, or aview, command is a graphical window on the status of an array. You can start it with the command aview and it displays a window similar to the one shown in Figure 2-5. The top window shows one line per node. There is a window for each node, headed by the node name and its hardware configuration. Each window contains a snapshot of the busiest processes in that node.

Figure 2-5. Typical Display from ArrayView (aview) Command

In an Array system you can start a program whose processes are in more than one node. In order to name such collections of processes, Array 3.0 software assigns each process to an array session handle (ASH).

An ASH is a number that is unique across the entire array (unlike a PID or PGID). An ASH is the same for every process that is part of a single array session—no matter which node the process runs in. You display and use ASH values with Array Services commands. Each time you log in to an Array node, your shell is given an ASH, which is used by all the processes you start from that shell.

The command ainfo ash returns the ASH of the current process on the local node, which is simply the ASH of the ainfo command itself.

homegrown 178% ainfo ash 
Array session handle of process 10068: 0x1111ffff000002c1
homegrown 179% ainfo ash 
Array session handle of process 10069: 0x1111ffff000002c1

In the preceding example, each instance of the ainfo command was a new process: first PID 10068, then PID 10069. However, the ASH is the same in both cases. This illustrates a very important rule: every process inherits its parent's ASH. In this case, each instance of array was forked by the command shell, and the ASH value shown is that of the shell, inherited by the child process.

You can create a new global ASH with the command ainfo newash, as follows:

homegrown 175% ainfo newash 
Allocating new global ASH
0x11110000308b2f7c

This feature has little use at present. There is no existing command that can change its ASH, so you cannot assign the new ASH to another command. It is possible to write a program that takes an ASH from a command-line option and uses the Array Services function setash() to change to that ASH (however such a program must be privileged). No such program is distributed with Array 3.0 (but see “Managing Array Service Handles”).

The command array ps returns a summary of all processes running on all nodes in an array. The display shows the ASH, the node, the PID, the associated username, the accumulated CPU time, and the command string.

To list all the processes on a particular node, use the -l and -s options. To list processes associated with a particular ASH, or a particular username, pipe the returned values through grep, as in the following example. (The display has been edited to save space.)

homegrown 182% array -l -s tokyo ps | fgrep wombat 
0x261cffff0000054c        tokyo 19007   wombat    0:00 -csh
0x261cffff0000054a        tokyo 17940   wombat    0:00 csh -c (setenv...
0x261cffff0000054c        tokyo 18941   wombat    0:00 csh -c (setenv...
0x261cffff0000054a        tokyo 17957   wombat    0:44 xem -geometry 84x42
0x261cffff0000054a        tokyo 17938   wombat    0:00 rshd
0x261cffff0000054a        tokyo 18022   wombat    0:00 /bin/csh -i
0x261cffff0000054a        tokyo 17980   wombat    0:03 /usr/gnu/lib/ema...
0x261cffff0000054c        tokyo 18928   wombat    0:00 rshd

When you have Performance Co-Pilot installed (see “Performance Co-Pilot”) you have two additional commands for listing processes: ashtop displays a continuously updated list of the processes that are executing under a specified ASH (see the ashtop(1) reference page, if installed). The arraytop command produces a similar display for the entire array (see the arraytop(1) reference page, if installed). Both of these, and additional features of Performance Co-Pilot, are described in the pcp_array(5) reference page.

The arshell command lets you start an arbitrary program on a single other node. The array command gives you the ability to suspend, resume, or kill all processes associated with a specified ASH.

Using arshell

The arshell command is an Array Services extension of the familiar rsh command; it executes a single IRIX command on a specified Array node. The difference from rsh is that the remote shell executes under the same ASH as the invoking shell (this is not true of simple rsh). The following example demonstrates the difference.

homegrown 179% ainfo ash Array session handle of process 8506: 0x1111ffff00000425 homegrown 180% rsh guest@tokyo ainfo ash Array session handle of process 13113: 0x261cffff0000145e homegrown 181% arshell guest@tokyo ainfo ash Array session handle of process 13119: 0x1111ffff00000425

You can use arshell to start a collection of unrelated programs in multiple nodes under a single ASH; then you can use the commands described under “Managing Session Processes” to stop, resume, or kill them.

Both MPI and PVM use arshell to start up distributed processes.

---

Tip: The shell is a process under its own ASH. If you use the array command to stop or kill all processes started from a shell, you will stop or kill the shell also. In order to create a group of programs under a single ASH that can be killed safely, proceed as follows:

---

1. Create a nested shell with a new ASH using newsess. Note the ASH value.

2. Within the new shell, start one or more programs using arshell.

3. Exit the nested shell.

Now you are back to the original shell. You know the ASH of all programs started from the nested shell. You can safely kill all jobs that have that ASH because the current shell is not affected.

#
# Local commands
#
command spin                    # Do nothing on multiple machines
        invoke /usr/lib/array/spin
        user    %USER
        group   %GROUP
        options nowait

#!/bin/sh
# Go into a tight loop
#
interrupted() {
        echo "spin has been interrupted - goodbye"
        exit 0
}
trap interrupted 1 2
while [ ! -f /tmp/spin.stop ]; do
        sleep 5
done
echo "spin has been stopped - goodbye"
exit 1

homegrown 175% ainfo ash
Array session handle of process 8912: 0x1111ffff0000032d
homegrown 176% newsess
homegrown 175% ainfo ash
Array session handle of process 8941: 0x11110000308b2fa6

homegrown 176% array spin

homegrown 177% exit
homegrown 178% homegrown 177% 
homegrown 177% ainfo ash
Array session handle of process 9257: 0x1111ffff0000032d
homegrown 179% array ps | fgrep 0x11110000308b2fa6
0x11110000308b2fa6  homegrown  9033   guest   0:00 /bin/sh /usr/lib/array/spin
0x11110000308b2fa6  homegrown  9618  guest  0:00 sleep 5
0x11110000308b2fa6        tokyo 26021  guest   0:00 /bin/sh /usr/lib/array/spin
0x11110000308b2fa6      tokyo 26072  guest  0:00 sleep 5
0x1111ffff0000032d  homegrown  9642  guest  0:00 fgrep 0x11110000308b2fa6

homegrown 180% array kill 0x11110000308b2fa6
homegrown 181% array ps | fgrep 0x11110000308b2fa6
0x1111ffff0000032d  homegrown 10030  guest  0:00 fgrep 0x11110000308b2fa6