Chapter 7. Array Services

Array Services includes administrator commands, libraries, daemons, and kernel extensions that support the execution of parallel applications across a number of hosts in a cluster, or array. The Message Passing Interface (MPI) uses Array Services to launch parallel applications. For information on MPI, see the Message Passing Toolkit: MPI Programmer's Manual.

The secure version of Array Services is built to make use of secure sockets layer (SSL) and secure shell (SSH).

Note: Differences between the standard version and the secure version of Array Services are noted throughout this chapter. For simplicity and clarity, the use of Array Services generally refers to both products. When noting differences between the two, a distinction is made between Array Services (AS), the standard product, and Secure Array Services (SAS), the security enhanced product.

To use the Array Services package on IRIX, you must have an Array Sessions enabled kernel. This accomplished when arsess kernel module is loaded at boot time. The arsess kernel module is listed in the /etc/sysconfig/kernel file. For more information, see “Installing Array Services”.

When using SAS, you also need to install the openssl package available from the IRIX distribution. For more information, see “Secure Array Services”.

A central concept in Array Services is the array session handle (ASH), a number that is used to logically group related processes that may be distributed across multiple systems. The ASH creates a global process namespace across the Array, facilitating accounting and administration

Array Services also provides an array configuration database, listing the nodes comprising an array. Array inventory inquiry functions provide a centralized, canonical view of the configuration of each node. Other array utilities let the administrator query and manipulate distributed array applications.

The Array Services package comprises the following primary components:

array daemon

Allocates ASH values and maintain information about node configuration and the relation of process IDs to ASHs. Array daemons reside on each node and work in cooperation.

array configuration database 

Describes the array configuration used by array daemons and user programs. One copy at each node.

ainfo command 

Lets the user or administrator query the Array configuration database and information about ASH values and processes.

array command 

Executes a specified command on one or more nodes. Commands are predefined by the administrator in the configuration database.

arshell command 

Starts a command remotely on a different node using the current ASH value.

aview command 

Displays a multiwindow, graphical display of each node's status.

libarray library 

Library of functions that allow user programs to call on the services of array daemons and the array configuration database.

The use of the ainfo, array, arshell, and aview commands is covered in “Using an Array”. The use of the libarray library is covered in“Array Services Library” .

This chapter covers the follow topics:

Installing Array Services

This section describes how to install Array Services and covers these topics:

Installing and Configuring Array Services for Single Host Systems

The normal IRIX installation process installs and pre-configures Array Services and Array Session module (arsess) software to enable single host Message Passing Toolkit (MPT) Message Passing Interface (MPI) jobs. The configuration steps encoded in the Array Services installation script also automatically issue the chkconfig (8) commands that register the Array Services arrayd(8) daemon to be started upon system reboot. If the usual system reboot is done after installing an IRIX overlay software release, you do not need to take any additional steps to configure Array Services.

Because there are two versions of the product, the standard version of Array Services is installed as sgi-arraysvcs. The security enhanced version is installed as sgi-sarraysvc. You cannot install both versions at the same time because they are mutually incompatible.

If you are installing a new Array Services on a live system, the Array Services daemon should be stopped before upgrading the software and then restarted after the upgrade. To stop the standard Array Services daemon, perform the following command:

% /etc/init.d/array stop

To stop the secure Array Services daemon, perform the following command:

% /etc/init.d/sarray stop

To start the standard Array Services daemon without having to reboot your system, perform the following command:

% /etc/init.d/array start

To start the sercure Array Services daemon without having to reboot your system, perform the following command:

% /etc/init.d/sarray start

The steps that are executed automatically by the Array Services at install time are described in the Array Services release notes and in “Automatic Array Serices Installation Steps”.

Installing and Configuring Array Services for Cluster or Partitioned Systems

On clustered or partitioned IRIX systems, it is often desirable to enable MPT MPI jobs to execute on multiple hosts, rather then being confined to a single host.

Note: If you run secure Array Services, you also need to install the openssl 0.9.7 package available from the overlay CD. In addition, for the steps that follow, keep in mind that the daemon for SAS is named sarrayd versus arrayd on standard Array Services.

To configure Array Services to execute on multiple hosts, perform the following:

  1. Identify a cluster name and a host list.

    Edit the /usr/lib/array/arrayd.conf file to list the machines in your cluster. The arrayd.conf file allows many specifications. For information about these specifications, see the arrayd.conf(4) man page. The only required specifications that need to be configured are the name for the cluster and a list of hostnames in the cluster.

    In the following steps, changes are made to the arrayd.conf file so that the cluster is given the name sgicluster and it consists of hosts named host1, host2, and so on:

    1. Add an array entry that lists the host names one per line, as follows:

      array sgicluster
                     machine host1
                     machine host2

    2. In the destination array directive, edit the default cluster name to be sgicluster, as follows:

      destination array sgicluster

  2. Choose an authentication policy:NONE or SIMPLE.

    You need to choose the security level under which Array Services will operate. The choices are authentication settings of NONE or SIMPLE. Either way, start by commenting out the line in /usr/lib/array/arrayd.auth file that reads AUTHENTICATION NOREMOTE. If no authentication is required at your site, uncomment the AUTHENTICATION NONE line in the arrayd.auth file. If you choose simple authentication, create an AUTHENTICATION SIMPLE section as described in the arrayd.auth(4) man page.

    Note: For sites concerned with security, AUTHENTICATION SIMPLE is a better choice. AUTHENTICATION is enforced to NONE when using secure Array Services and authentication is performed via certificate. For details, see “Secure Array Services”.

  3. When you are configuring Secure Array Services, you need to configure certificate. For information how how to do this, see “Secure Array Services Certificates”.

Automatic Array Serices Installation Steps

The following steps are performed automatically during installation of the Array Services:

  • An account must exist on all hosts in the array for the purpose of running certain Array Services commands. This is controlled by the /usr/lib/array/arrayd.conf configuration file. The default is to use the user account arraysvcs. The account name can be changed in arrayd.conf. The user account arraysvcs is installed by default.

  • The following entry is added to /etc/services file to define the arrayd service and port number. The default port number is 5434 and is specified in the arrayd.conf configuration file. Any value can be used for the port number, but all systems mentioned in the arrayd.conf file must use the same value.

    sgi-arrayd   5434/tcp    # SGI Array Services daemon

  • Standard Array Services are activated during installation with the chkconfig(1) command, as follows:

    chkconfig  --add array

    Secure Array Services are activated during installation with the chkconfig(1) command, as follows:

    chkconfig  --add sarray

Using an Array

An Array system is an aggregation of nodes, which are servers bound together with a high-speed network and Array Services 3.5 software. Array users have the advantage of greater performance and additional services. Array users access the system with familiar commands for job control, login and password management, and remote execution.

Array Services 3.5 augments conventional facilities with additional services for array users and for array administrators. The extensions include support for global session management, array configuration management, batch processing, message passing, system administration, and performance visualization.

This section introduces the extensions for Array use, with pointers to more detailed information. The main topics are as follows:

Using an Array System

The array system allows you to run distributed sessions on multiple nodes of an array. You can access the Array from either:

  • A workstation

  • An X terminal

  • An ASCII terminal

In each case, you log in to one node of the Array in the way you would log in to any remote UNIX host. From a workstation or an X terminal you can of course open more than one terminal window and log into more than one node.

Finding Basic Usage Information

In order to use an Array, you need the following items of information:

  • The name of the Array.

    You use this arrayname in Array Services commands.

  • The login name and password you will use on the Array.

    You use these when logging in to the Array to use it.

  • The hostnames of the array nodes.

    Typically these names follow a simple pattern, often arrayname1, arrayname2, and so on.

  • Any special resource-distribution or accounting rules that may apply to you or your group under a job scheduling system.

You can learn the hostnames of the array nodes if you know the array name, using the ainfo command as follows:

ainfo -a arrayname machines

Logging In to an Array

Each node in an Array has an associated hostname and IP network address. Typically, you use an Array by logging in to one node directly, or by logging in remotely from another host (such as the Array console or a networked workstation). For example, from a workstation on the same network, this command would log you in to the node named hydra6 as follows:

rlogin hydra6

For details of the rlogin command, see the rlogin(1) man page.

The system administrators of your array may choose to disallow direct node logins in order to schedule array resources. If your site is configured to disallow direct node logins, your administrators will be able to tell you how you are expected to submit work to the array-perhaps through remote execution software or batch queueing facilities.

Invoking a Program

Once you have access to an array, you can invoke programs of several classes:

  • Ordinary (sequential) applications

  • Parallel shared-memory applications within a node

  • Parallel message-passing applications within a node

  • Parallel message-passing applications distributed over multiple nodes (and possibly other servers on the same network running Array Services 3.5

If you are allowed to do so, you can invoke programs explicitly from a logged-in shell command line; or you may use remote execution or a batch queueing system.

Programs that are X Windows clients must be started from an X server, either an X Terminal or a workstation running X Windows.

Some application classes may require input in the form of command line options, environment variables, or support files upon execution. For example:

  • X client applications need the DISPLAY environment variable set to specify the X server (workstation or X-terminal) where their windows will display.

  • A multithreaded program may require environment variables to be set describing the number of threads.

    For example, C and Fortran programs that use parallel processing directives test the MP_SET_NUMTHREADS variable.

  • Message Passing Interface (MPI) and Parallel Virtual Machine (PVM) message-passing programs may require support files to describe how many tasks to invoke on specified nodes.

Some information sources on program invocation are listed in Table 7-1.

Table 7-1. Information Sources for Invoking a Program


Man Page



Remote login

rlogin (1)



Setting environment variables

environ (5), env(1)



Managing Local Processes

Each UNIX process has a process identifier (PID), a number that identifies that process within the node where it runs. It is important to realize that a PID is local to the node; so it is possible to have processes in different nodes using the same PID numbers.

Within a node, processes can be logically grouped in process groups. A process group is composed of a parent process together with all the processes that it creates. Each process group has a process group identifier (PGID). Like a PID, a PGID is defined locally to that node, and there is no guarantee of uniqueness across the Array.

Monitoring Local Processes and System Usage

You query the status of processes using the system command ps. To generate a full list of all processes on a local system, use a command such as the following:

ps -elfj

You can monitor the activity of processes using the command top (an ASCII display in a terminal window).

Scheduling and Killing Local Processes

You can start a process at a reduced priority, so that it interferes less with other processes, using the nice command. If you use the csh shell, specify /usr/bin/nice to avoid the built-in shell command nice. To start a whole shell at low priority, use a command like the one that follows:

/bin/nice /bin/sh

You can schedule commands to run at specific times using the at command. You can kill or stop processes using the kill command. To destroy the process with PID 13032, use a command such as the following:

kill -KILL 13032

Summary of Local Process Management Commands

Table 7-2, summarizes information about local process management.

Table 7-2. Information Sources: Local Process Management


Man Page

Process ID and process group


Listing and monitoring processes

ps(1), top(1)

Running programs at low priority

nice(1), batch(1)

Running programs at a scheduled time


Terminating a process


Using Array Services Commands

When an application starts processes on more than one node, the PID and PGID are no longer adequate to manage the application. The commands of Array Services 3.5 give you the ability to view the entire array, and to control the processes of multinode programs.

Note: You can use Array Services commands from any workstation connected to an array system. You don't have to be logged in to an array node.

The following commands are common to Array Services operations as shown in Table 7-3.

Note: The arshell(1) command is not installed or usable when you are running Secure Array Services


Table 7-3. Common Array Services Commands


Man Page

Array Services Overview

array_services (5)

ainfo command


array command

Use array (1); configuration: arrayd.conf(4)

arshell command

arshell (1)

aview command


newsess command

newsess (1)

About Array Sessions

Array Services is composed of a daemon-a background process that is started at boot time in every node-and a set of commands such as ainfo(1). The commands call on the daemon process in each node to get the information they need.

One concept that is basic to Array Services is the array session, which is a term for all the processes of one application, wherever they may execute. Normally, your login shell, with the programs you start from it, constitutes an array session. A batch job is an array session; and you can create a new shell with a new array session identity.

Each session is identified by an array session handle (ASH), a number that identifies any process that is part of that session. You use the ASH to query and to control all the processes of a program, even when they are running in different nodes.

About Names of Arrays and Nodes

Each node is server, and as such has a hostname. The hostname of a node is returned by the hostname(1) command executed in that node as follows:

% hostname

The command is simple and documented in the hostname(1) man page. The more complicated issues of hostname syntax, and of how hostnames are resolved to hardware addresses are covered in hostname(5).

An Array system as a whole has a name too. In most installations there is only a single Array, and you never need to specify which Array you mean. However, it is possible to have multiple Arrays available on a network, and you can direct Array Services commands to a specific Array.

About Authentication Keys

It is possible for the Array administrator to establish an authentication code, which is a 64-bit number, for all or some of the nodes in an array (see "Configuring Authentication Codes" on page 58). When this is done, each use of an Array Services command must specify the appropriate authentication key, as a command option, for the nodes it uses. Your system administrator will tell you if this is necessary.

Note: When running Secure Array Services, this configuration is not used. Authentication is enforced to AUTHENTICATION_NONE .

Summary of Common Command Options

The following Array Services commands have a consistent set of command options: ainfo(1), array(1), arshell(1), aview(1), and newsess(1). Table 7-4 is a summary of these options. Not all options are valid with all commands; and each command has unique options besides those shown. The default values of some options are set by environment variables listed in the next topic.

Note: The arshell(1) command is not installed or usable when you are running Secure Array Services.

Table 7-4. Array Services Command Option Summary


Used In


-a array

ainfo, array, aview

Specify a particular Array when more than one is accessible.


ainfo, array, arshell, aview

Send commands to other nodes directly, rather than through array daemon.


ainfo, array, arshell, aview

Forward commands to other nodes through the array daemon.

-Kl number

ainfo, array, aview

Authentication key (a 64-bit number) for the local node.

-Kr number

ainfo, array, aview

Authentication key (a 64-bit number) for the remote node.

-l (letter ell)

ainfo, array

Execute in context of the destination node, not necessarily the current node.

-l port

ainfo, array, arshell, aview

Nonstandard port number of array daemon.

-s hostname

ainfo, array, aview

Specify a destination node.

Specifying a Single Node

The -l and -s options work together. The -l (letter ell for “local”) option restricts the scope of a command to the node where the command is executed. By default, that is the node where the command is entered. When -l is not used, the scope of a query command is all nodes of the array. The -s (server, or node name) option directs the command to be executed on a specified node of the array. These options work together in query commands as follows:

  • To interrogate all nodes as seen by the local node, use neither option.

  • To interrogate only the local node, use only -l.

  • To interrogate all nodes as seen by a specified node, use only -s.

  • To interrogate only a particular node, use both -s and -l.

Common Environment Variables

The Array Services commands depend on environment variables to define default values for the less-common command options. These variables are summarized in Table 7-5.

Table 7-5. Array Services Environment Variables

Variable Name


Default When Undefined


When defined with a string starting with the letter y, all commands default to forwarding through the array daemon (option -F).

Commands default to direct communication (option -D).


The port (socket) number monitored by the array daemon on the destination node.

The standard number of 5434, or the number given with option -p.


Authentication key for the local node (option -Kl).

No authentication unless -Kl option is used.


Authentication key for the destination node (option -Kr).

No authentication unless -Kr option is used.


The destination node, when not specified by the -s option.

The local node, or the node given with -s.

Interrogating the Array

Any user of an Array system can use Array Services commands to check the hardware components and the software workload of the Array. The commands needed are ainfo, array, and aview.

Learning Array Names

If your network includes more than one Array system, you can use ainfo arrays at one array node to list all the Array names that are configured, as in the following example.

homegrown% ainfo arrays
Arrays known to array services daemon
ARRAY DevArray
    IDENT 0x3381
ARRAY BigDevArray
    IDENT 0x7456
ARRAY test
    IDENT 0x655e

Array names are configured into the array database by the administrator. Different Arrays might know different sets of other Array names.

Learning Node Names

You can use ainfo machines to learn the names and some features of all nodes in the current Array, as in the following example.

homegrown 175% ainfo -b machines 
machine homegrown homegrown 5434 0
machine disarray disarray 5434 0
machine datarray datarray 5434 0
machine tokyo tokyo 5434 0

In this example, the -b option of ainfo is used to get a concise display.

Learning Node Features

You can use ainfo nodeinfo to request detailed information about one or all nodes in the array. To get information about the local node, use ainfo -l nodeinfo. However, to get information about only a particular other node, for example node tokyo, use -l and -s, as in the following example. (The example has been edited for brevity.)

homegrown 181% ainfo -s tokyo -l nodeinfo
Node information for server on machine "tokyo"
    VERSION  1.2
        BOARD: TYPE 15   SPEED 190
            CPU:   TYPE 9   REVISION 2.4
            FPU:   TYPE 9   REVISION 0.0
    16 IP INTERFACES  HOSTNAME tokyo   HOSTID 0xc01a5035
        DEVICE  et0    NETWORK    ADDRESS  UP
        DEVICE atm0    NETWORK    ADDRESS  UP
        DEVICE atm1    NETWORK    ADDRESS  UP
        512 MB MAIN MEMORY
        INTERLEAVE 4

If the -l option is omitted, the destination node will return information about every node that it knows.

Learning User Names and Workload

The system commands who(1), top(1), and uptime(1) are commonly used to get information about users and workload on one server. The array(1) command offers Array-wide equivalents to these commands.

Learning User Names

To get the names of all users logged in to the whole array, use array who. To learn the names of users logged in to a particular node, for example tokyo, use -l and -s, as in the following example. (The example has been edited for brevity and security.)

 homegrown 180% array -s tokyo -l who 
joecd    tokyo        frummage.eng.sgi -tcsh 
joecd    tokyo        frummage.eng.sgi -tcsh 
benf     tokyo        einstein.ued.sgi. /bin/tcsh 
yohn     tokyo vi +153 fs/procfs/prd

Learning Workload

Two variants of the array command return workload information. The array-wide equivalent of uptime is array uptime, as follows:

homegrown 181% array uptime 
   homegrown:  up 1 day,  7:40,  26 users,  load average: 7.21, 6.35, 4.72
    disarray:  up  2:53,  0 user,  load average: 0.00, 0.00, 0.00
    datarray:  up  5:34,  1 user,  load average: 0.00, 0.00, 0.00
       tokyo:  up 7 days,  9:11,  17 users,  load average: 0.15, 0.31, 0.29
homegrown 182% array -l -s tokyo uptime 
       tokyo:  up 7 days,  9:11,  17 users,  load average: 0.12, 0.30, 0.28

The command array top lists the processes that are currently using the most CPU time, with their ASH values, as in the following example.

homegrown 183% array top 
        ASH        Host           PID User       %CPU Command
0x1111ffff00000000 homegrown        5 root       1.20 vfs_sync
0x1111ffff000001e9 homegrown     1327 guest      1.19 atop
0x1111ffff000001e9 tokyo        19816 guest      0.73 atop
0x1111ffff000001e9 disarray      1106 guest      0.47 atop
0x1111ffff000001e9 datarray      1423 guest      0.42 atop
0x1111ffff000000c0 homegrown    29683 kchang     0.37 ld
0x1111ffff0000001e homegrown     1324 root       0.17 arrayd
0x1111ffff00000000 homegrown      229 root       0.14 routed
0x1111ffff00000000 homegrown       19 root       0.09 pdflush
0x1111ffff000001e9 disarray      1105 guest      0.02 atopm

The -l and -s options can be used to select data about a single node, as usual.

Browsing With ArrayView

The ArrayView window shows the status of an array. You can start it with the command aview and it displays a window similar to the one shown in Figure 7-1. The top window shows one line per node. There is a window for each node, headed by the node name and its hardware configuration. Each window contains a snapshot of the busiest processes in that node.

Figure 7-1. Typical Display from ArrayView

Typical Display from ArrayView

Managing Distributed Processes

Using commands from Array Services 3.5, you can create and manage processes that are distributed across multiple nodes of the Array system.

About Array Session Handles (ASH)

In an Array system you can start a program with processes that are in more than one node. In order to name such collections of processes, Array Services 3.5 software assigns each process to an array session handle (ASH).

An ASH is a number that is unique across the entire array (unlike a PID or PGID). An ASH is the same for every process that is part of a single array session--no matter which node the process runs in. You display and use ASH values with Array Services commands. Each time you log in to an Array node, your shell is given an ASH, which is used by all the processes you start from that shell.

The command ainfo ash returns the ASH of the current process on the local node, which is simply the ASH of the ainfo command itself.

homegrown 178% ainfo ash 
Array session handle of process 10068: 0x1111ffff000002c1
homegrown 179% ainfo ash 
Array session handle of process 10069: 0x1111ffff000002c1

In the preceding example, each instance of the ainfo command was a new process: first PID 10068, then PID 10069. However, the ASH is the same in both cases. This illustrates a very important rule: every process inherits its parent's ASH. In this case, each instance of array was forked by the command shell, and the ASH value shown is that of the shell, inherited by the child process.

You can create a new global ASH with the command ainfo newash, as follows:

homegrown 175% ainfo newash 
Allocating new global ASH

This feature has little use at present. There is no existing command that can change its ASH, so you cannot assign the new ASH to another command. It is possible to write a program that takes an ASH from a command-line option and uses the Array Services function setash() to change to that ASH (however such a program must be privileged). No such program is distributed with Array Services 3.5 (but see “Managing Array Service Handles”).

Listing Processes and ASH Values

The command array ps returns a summary of all processes running on all nodes in an array. The display shows the ASH, the node, the PID, the associated username, the accumulated CPU time, and the command string.

To list all the processes on a particular node, use the -l and -s options. To list processes associated with a particular ASH, or a particular username, pipe the returned values through grep, as in the following example. (The display has been edited to save space.)

homegrown 182% array -l -s tokyo ps | fgrep wombat 
0x261cffff0000054c        tokyo 19007   wombat    0:00 -csh
0x261cffff0000054a        tokyo 17940   wombat    0:00 csh -c (setenv...
0x261cffff0000054c        tokyo 18941   wombat    0:00 csh -c (setenv...
0x261cffff0000054a        tokyo 17957   wombat    0:44 xem -geometry 84x42
0x261cffff0000054a        tokyo 17938   wombat    0:00 rshd
0x261cffff0000054a        tokyo 18022   wombat    0:00 /bin/csh -i
0x261cffff0000054a        tokyo 17980   wombat    0:03 /usr/gnu/lib/ema...
0x261cffff0000054c        tokyo 18928   wombat    0:00 rshd

Controlling Processes

The arshell command lets you start an arbitrary program on a single other node. The array command gives you the ability to suspend, resume, or kill all processes associated with a specified ASH.

Using arshell

The arshell command is an Array Services extension of the familiar rsh command; it executes a single system command on a specified Array node. The difference from rsh is that the remote shell executes under the same ASH as the invoking shell (this is not true of simple rsh). The following example demonstrates the difference.

homegrown 179% ainfo ash
Array session handle of process 8506: 0x1111ffff00000425
homegrown 180% rsh [email protected] ainfo ash
Array session handle of process 13113: 0x261cffff0000145e
homegrown 181% arshell [email protected] ainfo ash
Array session handle of process 13119: 0x1111ffff00000425

You can use arshell to start a collection of unrelated programs in multiple nodes under a single ASH; then you can use the commands described under “Managing Session Processes” to stop, resume, or kill them.

Both MPI and PVM use arshell to start up distributed processes.

Tip: The shell is a process under its own ASH. If you use the array command to stop or kill all processes started from a shell, you will stop or kill the shell also. In order to create a group of programs under a single ASH that can be killed safely, proceed as follows:
  1. Create a nested shell with a new ASH using newsess. Note the ASH value.

  2. Within the new shell, start one or more programs using arshell.

  3. Exit the nested shell.

Now you are back to the original shell. You know the ASH of all programs started from the nested shell. You can safely kill all jobs that have that ASH because the current shell is not affected.

About the Distributed Example

The programs launched with arshell are not coordinated (they could of course be written to communicate with each other, for example using sockets), and you must start each program individually.

The array command is designed to permit the simultaneous launch of programs on all nodes with a single command. However, array can only launch programs that have been configured into it, in the Array Services configuration file. (The creation and management of this file is discussed under “About Array Configuration”.)

In order to demonstrate process management in a simple way from the command line, the following command was inserted into the configuration file /usr/lib/array/arrayd.conf:

# Local commands
command spin                    # Do nothing on multiple machines
        invoke /usr/lib/array/spin
        user    %USER
        group   %GROUP
        options nowait

The invoked command, /usr/lib/array/spin, is a shell script that does nothing in a loop, as follows:

# Go into a tight loop
interrupted() {
        echo "spin has been interrupted - goodbye"
        exit 0
trap interrupted 1 2
while [ ! -f /tmp/spin.stop ]; do
        sleep 5
echo "spin has been stopped - goodbye"
exit 1

With this preparation, the command array spin starts a process executing that script on every processor in the array. Alternatively, array -l -s nodename spin would start a process on one specific node.

Managing Session Processes

The following command sequence creates and then kills a spin process in every node. The first step creates a new session with its own ASH. This is so that later, array kill can be used without killing the interactive shell.

homegrown 175% ainfo ash
Array session handle of process 8912: 0x1111ffff0000032d
homegrown 176% newsess
homegrown 175% ainfo ash
Array session handle of process 8941: 0x11110000308b2fa6

In the new session with ASH 0x11110000308b2fa6, the command array spin starts the /usr/lib/array/spin script on every node. In this test array, there were only two nodes on this day, homegrown and tokyo.

homegrown 176% array spin

After exiting back to the original shell, the command array ps is used to search for all processes that have the ASH 0x11110000308b2fa6.

homegrown 177% exit
homegrown 178% homegrown 177% 
homegrown 177% ainfo ash
Array session handle of process 9257: 0x1111ffff0000032d
homegrown 179% array ps | fgrep 0x11110000308b2fa6
0x11110000308b2fa6  homegrown  9033   guest   0:00 /bin/sh /usr/lib/array/spin
0x11110000308b2fa6  homegrown  9618  guest  0:00 sleep 5
0x11110000308b2fa6        tokyo 26021  guest   0:00 /bin/sh /usr/lib/array/spin
0x11110000308b2fa6      tokyo 26072  guest  0:00 sleep 5
0x1111ffff0000032d  homegrown  9642  guest  0:00 fgrep 0x11110000308b2fa6

There are two processes related to the spin script on each node. The next command kills them all.

homegrown 180% array kill 0x11110000308b2fa6
homegrown 181% array ps | fgrep 0x11110000308b2fa6
0x1111ffff0000032d  homegrown 10030  guest  0:00 fgrep 0x11110000308b2fa6

The command array suspend 0x11110000308b2fa6 would suspend the processes instead (however, it is hard to demonstrate that a sleep command has been suspended).

About Job Container IDs

Array systems that are running IRIX 6.5.7f (or later) and Array Services version 3.4 (or later) have the capability to forward job IDs (JIDs) from the initiating host. All of the processes running in the ASH across one or more nodes in an array also belong to the same job. For a complete description of the job container and it usage, see the job_limits(5) man page or IRIX Admin: Resource Administration.

When processes are running on the initiating host, they belong to the same job as the initiating process and operate under the limits established for that job. On remote nodes, a new job is created using the same JID as the initiating process. Job limits for a job on remote nodes use the systune defaults and are set using the systune(1M) command on the initiating host.

About Array Configuration

The system administrator has to initialize the Array configuration database, a file that is used by the Array Services daemon in executing almost every ainfo and array command. For details about array configuration, see the man pages cited in Table 7-6.

Table 7-6. Information Sources: Array Configuration


Man Page

Array Services overview

array_services (5)  

Array Services user commands

ainfo(1) , array(1)  

Array Services daemon overview

arrayd (1m)  

Configuration file format

arrayd.conf (4) , /usr/lib/array/arrayd.conf.template  

Configuration file validator

ascheck (1)  

Array Services simple configurator

arrayconfig (1m)

Security Considerations for Standard Array Services

The array services daemon, arrayd(1M), runs as root. As with other system services, if it is configured carelessly it is possible for arbitrary and possibly unauthorized user to disrupt or even damage a running system.

By default, most array commands are executed using the user, group, and project ID of either the user that issued the original command, or arraysvcs. When adding new array commands to arrayd.conf , or modifying existing ones, always use the most restrictive IDs possible in order to minimize trouble if a hostile or careless user were to run that command. Avoid adding commands that run with "more powerful" IDs (such as user "root" or group "sys") than the user. If such commands are necessary, analyze them carefully to ensure that an arbitrary user would not be granted any more privileges than expected, much the same as one would analyze a setuid program.

In the default array services configuration, the arrayd daemon allows all the local requests to access arrayd but not the remote requests. In order to let the remote requests access the arrayd, the AUTHENTICATION parameter needs to be set to NONE in the /usr/lib/array/arrayd.auth file. By default it is set to NOREMOTE. When the AUTHENTICATION parameter is set to NONE, the arrayd daemon assumes that a remote user will accurately identify itself when making a request. In other words, if a request claims to be coming from user "abc", the arrayd daemon assumes that it is in fact from user "abc" and not somebody spoofing "abc". This should be adequate for systems that are behind a network firewall or otherwise protected from hostile attack, and in which all the users inside the firewall are presumed to be non-hostile. On systems, for which this is not the case (for example, those that are attached to a public network, or when individual machines otherwise cannot be trusted), the Array Services AUTHENTICATION parmeter should be set to NOREMOTE. When AUTHENTICATION is set to NONE, all requests from remote systems are authenticated using a mechanism that involves private keys that are known only to the super-users on the local and remote systems. Requests originating on systems that do not have these private keys are rejected. For more details, see the section on "Authentication Information" in the arrayd.conf(4) man page.

The arrayd daemon does not support mapping user, group or project names between two different namespaces; all members of an array are assumed to share the same namespace for users, groups, and projects. Thus, if systems "A" and "B" are members of the same array, username "abc" on system A is assumed to be the same user as username "abc" on system B. This is most significant in the case of username "root". Authentication should be used if necessary to prevent access to an array by machines using a different namespace.

About the Uses of the Configuration File

The configuration files are read by the Array Services daemon when it starts. Normally it is started in each node during the system startup. (You can also run the daemon from a command line in order to check the syntax of the configuration files.)

The configuration files contain data needed by ainfo and array:

  • The names of Array systems, including the current Array but also any other Arrays on which a user could run an Array Services command (reported by ainfo).

  • The names and types of the nodes in each named Array, especially the hostnames that would be used in an Array Services command (reported by ainfo).

  • The authentication keys, if any, that must be used with Array Services commands (required as -Kl and -Kr command options, see “Summary of Common Command Options”).

  • The commands that are valid with the array command.

About Configuration File Format and Contents

A configuration file is a readable text file. The file contains entries of the following four types, which are detailed in later topics.

Array definition 

Describes this array and other known arrays, including array names and the node names and types.

Command definition 

Specifies the usage and operation of a command that can be invoked through the array command.


Specifies authentication numbers that must be used to access the Array.

Local option 

Options that modify the operation of the other entries or arrayd.

Blank lines, white space, and comment lines beginning with “#” can be used freely for readability. Entries can be in any order in any of the files read by arrayd.

Besides punctuation, entries are formed with a keyword-based syntax. Keyword recognition is not case-sensitive; however keywords are shown in uppercase in this text and in the man page. The entries are primarily formed from keywords, numbers, and quoted strings, as detailed in the man page arrayd.conf(4) .

Loading Configuration Data

The Array Services daemon, arrayd, can take one or more filenames as arguments. It reads them all, and treats them like logical continuations (in effect, it concatenates them). If no filenames are specified, it reads /usr/lib/array/arrayd.conf and /usr/lib/array/arrayd.auth. A different set of files, and any other arrayd command-line options, can be written into the file /etc/config/arrayd.options, which is read by the startup script that launches arrayd at boot time.

Since configuration data can be stored in two or more files, you can combine different strategies, for example:

  • One file can have different access permissions than another. Typically, /usr/lib/array/arrayd.conf is world-readable and contains the available array commands, while /usr/lib/array/arrayd.auth is readable only by root and contains authentication codes.

  • One node can have different configuration data than another. For example, certain commands might be defined only in certain nodes; or only the nodes used for interactive logins might know the names of all other nodes.

  • You can use NFS-mounted configuration files. You could put a small configuration file on each machine to define the Array and authentication keys, but you could have a larger file defining array commands that is NFS-mounted from one node.

After you modify the configuration files, you can make arrayd reload them by killing the daemon and restarting it in each machine. The script /etc/init.d/array supports this operation:

To kill daemon, execute this command:

/etc/init.d/array stop

To kill and restart the daemon in one operation; peform the following command:

/etc/init.d/array restart

The Array Services daemon in any node knows only the information in the configuration files available in that node. This can be an advantage, in that you can limit the use of particular nodes; but it does require that you take pains to keep common information synchronized. (An automated way to do this is summarized under “Designing New Array Commands”.)

About Substitution Syntax

The man page arrayd.conf(4) details the syntax rules for forming entries in the configuration files. An important feature of this syntax is the use of several kinds of text substitution, by which variable text is substituted into entries when they are executed.

Most of the supported substitutions are used in command entries. These substitutions are performed dynamically, each time the array command invokes a subcommand. At that time, substitutions insert values that are unique to the invocation of that subcommand. For example, the value %USER inserts the user ID of the user who is invoking the array command. Such a substitution has no meaning except during execution of a command.

Substitutions in other configuration entries are performed only once, at the time the configuration file is read by arrayd. Only environment variable substitution makes sense in these entries. The environment variable values that are substituted are the values inherited by arrayd from the script that invokes it, which is /etc/init.d/array.

Testing Configuration Changes

The configuration files contain many sections and options (detailed in the section that follow this one). The Array Services command ascheck performs a basic sanity check of all configuration files in the array.

After making a change, you can test an individual configuration file for correct syntax by executing arrayd as a command with the -c and -f options. For example, suppose you have just added a new command definition to /usr/lib/array/arrayd.local. You can check its syntax with the following command:

arrayd -c -f /usr/lib/array/arrayd.local

When testing new commands for correct operation, you need to see the warning and error messages produced by arrayd and processes that it may spawn. The stderr messages from a daemon are not normally visible. You can make them visible by the following procedure:

  1. On one node, kill the daemon.

  2. In one shell window on that node, start arrayd with the options -n -v. Instead of moving into the background, it remains attached to the shell terminal.

    Note: Although arrayd becomes functional in this mode, it does not refer to /etc/config/arrayd.options, so you need to specify explicitly all command-line options, such as the names of nonstandard configuration files.

  3. From another shell window on the same or other nodes, issue ainfo and array commands to test the new configuration data. Diagnostic output appears in the arrayd shell window.

  4. Terminate arrayd and restart it as a daemon (without -n).

During steps 1, 2, and 4, the test node may fail to respond to ainfo and array commands, so users should be warned that the Array is in test mode.

Configuring Arrays and Machines

Each ARRAY entry gives the name and composition of an Array system that users can access. At least one ARRAY must be defined at every node, the array in use.

Note: ARRAY is a keyword.

Specifying Arrayname and Machine Names

A simple example of an ARRAY definition is a follows:

array simple
        machine congo
        machine niger
        machine nile

The arrayname simple is the value the user must specify in the -a option (see “Summary of Common Command Options”). One arrayname should be specified in a DESTINATION ARRAY local option as the default array (reported by ainfo dflt). Local options are listed under “Configuring Local Options”.

It is recommended that you have at least one array called me that just contains the localhost. The default arrayd.conf file has the me array defined as the default destination array.

The MACHINE subentries of ARRAY define the nodenames that the user can specify with the -s option. These names are also reported by the command ainfo machines.

Specifying IP Addresses and Ports

The simple MACHINE subentries shown in the example are based on the assumption that the hostname is the same as the machine's name to Domain Name Services (DNS). If a machine's IP address cannot be obtained from the given hostname, you must provide a HOSTNAME subentry to specify either a completely qualified domain name or an IP address, as follows:

array simple
        machine congo
            port 8820
        machine niger
        machine nile
            hostname ""

The preceding example also shows how the PORT subentry can be used to specify that arrayd in a particular machine uses a different socket number than the default 5434.

Specifying Additional Attributes

Under both ARRAY and MACHINE you can insert attributes, which are named string values. These attributes are not used by Array Services, but they are displayed by ainfo and can be returned to programs using the Array Services library (see “Array Services Library”). Some examples of attributes would be as follows:

array simple
       array_attribute config_date="04/03/96"
       machine a_node
       machine_attribute aka="congo"

Tip: You can write code that fetches any arrayname, machine name, or attribute string from any node in the array. See “Database Interrogation”.

Configuring Authentication Codes

In Array Services 3.5 only one type of authentication is provided: a simple numeric key that can be required with any Array Services command. You can specify a single authentication code number for each node. The user must specify the code with any command entered at that node, or addressed to that node using the -s option (see “Summary of Common Command Options”).

The arshell command is like rsh in that it runs a command on another machine under the userid of the invoking user. Use of authentication codes makes Array Services somewhat more secure than rsh.

Configuring Array Commands

The user can invoke arbitrary system commands on single nodes using the arshell command (see “Using arshell”). The user can also launch MPI and PVM programs that automatically distribute over multiple nodes. However, the only way to launch coordinated system programs on all nodes at once is to use the array command. This command does not accept any system command; it only permits execution of commands that the administrator has configured into the Array Services database.

You can define any set of commands that your users need. You have complete control over how any single Array node executes a command (the definition can be different in different nodes). A command can simply invoke a standard system command, or, since you can define a command as invoking a script, you can make a command arbitrarily complex.

Operation of Array Commands

When a user invokes the array command, the subcommand and its arguments are processed by the destination node specified by -s. Unless the -l option was given, that daemon also distributes the subcommand and its arguments to all other array nodes that it knows about (the destination node might be configured with only a subset of nodes). At each node, arrayd searches the configuration database for a COMMAND entry with the same name as the array subcommand.

In the following example, the subcommand uptime is processed by arrayd in node tokyo:

array -s tokyo uptime

When arrayd finds the subcommand valid, it distributes it to every node that is configured in the default array at node tokyo.

The COMMAND entry for uptime is distributed in this form (you can read it in the file /usr/lib/array/arrayd.conf ).

command uptime          # Display uptime/load of all nodes in array
        invoke /usr/lib/array/auptime %LOCAL

The INVOKE subentry tells arrayd how to execute this command. In this case, it executes a shell script /usr/lib/array/auptime , passing it one argument, the name of the local node. This command is executed at every node, with %LOCAL replaced by that node's name.

Summary of Command Definition Syntax

Look at the basic set of commands distributed with Array Services 3.5 (/usr/lib/array/arrayd.conf ). Each COMMAND entry is defined using the subentries shown in Table 7-7. (These are described in great detail in the man page arrayd.conf(4).)

Table 7-7. Subentries of a COMMAND Definition


Meaning of Following Values


The name of the command as the user gives it to array.


A system command to be executed on every node. The argument values can be literals, or arguments given by the user, or other substitution values.


A system command to be executed only on the distributing node, to gather the streams of output from all nodes and combine them into a single stream.


The user ID under which the INVOKE and MERGE commands run. Usually given as USER %USER, so as to run as the user who invoked array.


The group name under which the INVOKE and MERGE commands run. Usually given as GROUP %GROUP, so as to run in the group of the user who invoked array (see the groups(1) man page).


The project under which the INVOKE and MERGE commands run. Usually given as PROJECT %PROJECT, so as to run in the project of the user who invoked array (see the projects(5) man page).


A variety of options to modify this command; see Table 7-9.

The system commands called by INVOKE and MERGE must be specified as full pathnames, because arrayd has no defined execution path. As with a shell script, these system commands are often composed from a few literal values and many substitution strings. The substitutions that are supported (which are documented in detail in the arrayd.conf(4)  man page) are summarized in Table 7-8.

Table 7-8. Substitutions Used in a COMMAND Definition


Replacement Value

%1..%9; %ARG(n); %ALLARGS; %OPTARG(n)

Argument tokens from the user's subcommand. %OPTARG does not produce an error message if the specified argument is omitted.


The effective user ID, effective group ID, and project of the user who invoked array.


The real user ID and real group ID of the user who invoked array.


The ASH under which the INVOKE or MERGE command is to run.


List of PID values for a specified ASH. %PID(%ASH) is a common use.


The array name, either default or as given in the -a option.


The hostname of the executing node.


The full domain name of the node where the array command ran and the output is to be viewed.


List of names of temporary files, each containing the output from one node's INVOKE command (valid only in the MERGE subentry).

The OPTIONS subentry permits a number of important modifications of the command execution; these are summarized in Table 7-9.

Table 7-9. Options of the COMMAND Definition


Effect on Command


Do not distribute to other nodes (effectively forces the -l option).


Execute the INVOKE command under a newly created ASH. %ASH in the INVOKE line is the new ASH. The MERGE command runs under the original ASH, and %ASH substitutes as the old ASH in that line.


Set both the real and effective user ID from the USER subentry (normally USER only sets the effective UID).


Set both the real and effective group ID from the GROUP subentry (normally GROUP sets only the effective GID).


Discard the output of INVOKE, unless a MERGE subentry is given. If a MERGE subentry is given, pass INVOKE output to MERGE as usual and discard the MERGE output.


Discard the output and return as soon as the processes are invoked; do not wait for completion (a MERGE subentry is ineffective).

Configuring Local Options

The LOCAL entry specifies options to arrayd itself. The most important options are summarized in Table 7-10.

Table 7-10. Subentries of the LOCAL Entry




Pathname for the arrayd working directory, which is the initial, current working directory of INVOKE and MERGE commands. The default is /usr/lib/array .


Name of the default array, used when the user omits the -a option. When only one ARRAY entry is given, it is the default destination.


Default values for COMMAND execution when USER, GROUP, or PROJECT are omitted from the COMMAND definition.


Value returned in this node by %LOCAL. Default is the hostname.


Socket to be used by arrayd.

If you do not supply LOCAL USER, GROUP, and PROJECT values, the default values for USER and GROUP are “guest.”

The HOSTNAME entry is needed whenever the hostname command does not return a node name as specified in the ARRAY MACHINE entry. In order to supply a LOCAL HOSTNAME entry unique to each node, each node needs an individualized copy of at least one configuration file.

Designing New Array Commands

A basic set of commands is distributed in the file /usr/lib/array/arrayd.conf.template . You should examine this file carefully before defining commands of your own. You can define new commands which then become available to the users of the Array system.

Typically, a new command will be defined with an INVOKE subentry that names a script written in sh, csh, or Perl syntax. You use the substitution values to set up arguments to the script. You use the USER, GROUP, PROJECT, and OPTIONS subentries to establish the execution conditions of the script. For one example of a command definition using a simple script, see “About the Distributed Example”.

Within the invoked script, you can write any amount of logic to verify and validate the arguments and to execute any sequence of commands. For an example of a script in Perl, see /usr/lib/array/aps, which is invoked by the array ps command.

Note: Perl is a particularly interesting choice for array commands, since Perl has native support for socket I/O. In principle at least, you could build a distributed application in Perl in which multiple instances are launched by array and coordinate and exchange data using sockets. Performance would not rival the highly tuned MPI and PVM libraries, but development would be simpler.

The administrator has need for distributed applications as well, since the configuration files are distributed over the Array. Here is an example of a distributed command to reinitialize the Array Services database on all nodes at once. The script to be executed at each node, called /usr/lib/array/arrayd-reinit would read as follows:

# Script to reinitialize arrayd with a new configuration file
# Usage:  arrayd-reinit <hostname:new-config-file>
sleep 10      # Let old arrayd finish distributing
rcp $1 /usr/lib/array/
/etc/init.d/array restart
exit 0

The script uses rcp to copy a specified file (presumably a configuration file such as arrayd.conf) into /usr/lib/array (this will fail if %USER is not privileged). Then the script restarts arrayd (see /etc/init.d/array) to reread configuration files.

The command definition would be as follows:

command reinit
   invoke /usr/lib/array/arrayd-reinit %ORIGIN:%1
   user   %USER
   group  %GROUP
   options nowait   # Exit before restart occurs!

The INVOKE subentry calls the restart script shown above. The NOWAIT option prevents the daemon's waiting for the script to finish, since the script will kill the daemon.

Array Services Library

Array Services consists of a configuration database, a daemon ( arrayd) that runs in each node to provide services, and several user-level commands. The facilities of Array Services are also available to developers through the Array Services library, a set of functions through which you can interrogate the configuration database and call on the services of arrayd.

The commands of Array Services are covered in “Using Array Services Commands”. The administration of Array Services is described in “About Array Configuration”, and the sections that follow it. These sections provide useful background information for understanding the Array Services library.

The programming interface to Array Services is declared in the header file /usr/include/arraysvcs.h. The object code is located in /usr/lib/, included in a program by specifying -larray during compilation. The library is distributed in o32, n32, and 64-bit versions on IRIX (not all need to be installed).

The library functions can be grouped into these categories:

  • Functions to connect to Array Services daemons in the local or other nodes, and to get and set arrayd options.

  • Functions to interrogate the Array Services configuration database, listing arrays, nodes, and attributes of arrays and nodes.

  • Functions to allocate Array Session Handles (ASHs), to query active ASHs, and to change the relationship between PIDs and ASHs.

  • A function to execute a command for the array command (see “Operation of Array Commands”).

  • A function to execute any arbitrary user command on an array node.

These functions are examined in following sections.

Data Structures

The Array Services functions work with a number of data structures that are declared in arraysvcs.h. In general, each data structure is allocated by one particular function, which returns a pointer to the structure as the function's result. Your code uses the returned structure, possibly passing it as an argument to other functions.

When your code is finished with a structure, it is expected to call a specific function that frees that type of structure. If your code does not free each structure, a memory leak results.

The data structures and their contents are summarized in Table 7-11.

Table 7-11. Array Services Data Structures



Freed By Function


Name and attributes of an Array.



List of asarray_t structures.



List of ASH values.



Describes output of executing an array command on one node, including temporary files and socket numbers.

Freed as part of a list


List of command results, one ascmdrslt_t per node where an array command was executed.



Configuration data about one node: machine name and attributes.

Freed as part of a list


List of asmachine_t structures, one per machine in the queried array



List of PID values.


Error Message Conventions

The functions of the Array Services library have a complicated convention for error return codes. The man pages related to this convention are listed in Table 7-12.

Table 7-12. Error Message Functions



aserrorcode (3X)  

Discusses the error code conventions and some macro functions used to extract subfields from an error code.

asmakeerror (3X)  

Constructs an error code value from its component parts.

asstrerror (3X)  

Returns a descriptive string for a given error code value.

asperror (3X)  

Prints a descriptive string, with a specified heading string, on stderr.

In general, each function sets a value in the global aserrorcode structure, which has type aserror_t (not necessarily an int). An error code is a structured value with these parts:

  • aserrno is a general error number similar to those declared in sys/errno.h.

  • aserrwhy documents the cause of the error.

  • aserrwhat documents the component that detected the error.

  • aserrextra may give additional information.

Macro functions to extract these subfields from the global aserrorcode structure are provided.

Connecting to Array Services Daemons

The functions listed in Table 7-13 are used to open a connection between the node where your program runs and an instance of arrayd in the same or another node.

Table 7-13. Functions for Connections to Array Services Daemons



asopenserver (3X)  

Establishes a logical connection to arrayd in a specified node, returning a token that represents that connection for use in other functions.

ascloseserver (3X)  

Closes an arrayd connection created by asopenserver().

asgetserveropt (3X)  

Returns the local options currently in use by an instance of arrayd.

asdfltserveropt (3X)  

Returns the default options in effect at an instance of arrayd.

assetserveropt (3X)  

Sets new options for an instance of arrayd.

The key function is asopenserver(). It takes a node name as a character string (as a user would give it in the -s option; see “Summary of Common Command Options”), and optionally a socket number to override the default arrayd socket number. This function opens a socket connection to the specified instance of arrayd. The returned token (type asserver_t ) stands for that connection and is passed to other functions.

The functions for getting and setting server options can change the configured options shown in Table 7-14. To set these options is the programmatic equivalent of passing command line options in an Array Services command (see “About Array Configuration”, and “Using Array Services Commands”).

Table 7-14. Server Options That Functions Can Query or Change






Timeout interval for any request to this server



Timeout interval for connecting to this server



Whether or not Array Services requests should be forwarded through the local arrayd or sent directly (using the -F option)



The local authentication key (the -Kl command option)



The remote authentication key ( -Kr command option)



In default options only, the default socket number



The hostname for this connection

Database Interrogation

The functions summarized in Table 7-15 are used to interrogate the configuration database used by arrayd in a specified node (see “About Array Configuration”).

Table 7-15. Functions for Interrogating the Configuration



asgetdfltarray (3X)

Returns the array name and all attribute strings for the default array known to a specified server in an asarray_t structure

aslistarrays (3X)

Returns the names of all arrays, with their attribute strings, from a specified server as an asarraylist_t structure

aslistmachines (3X)

Returns the names of all machines, with their attribute strings, from a specified server as an asmachinelist_t structure

asgetattr (3X)

Searches for a particular attribute name in a list of attribute strings and return its value

Using these functions you can extract any array name, node name, or attribute that is known to an arrayd instance you have opened.

Managing Array Service Handles

The functions summarized in Table 7-16 are used to create and interrogate ASH values.

Table 7-16. Functions for Managing Array Service Handles



asallocash (3X)

Allocates a new ASH value. The value is only created, it is not applied to any process.

aspidsinash (3X)

Returns a list of PID values associated with an ASH at a specified server, as an aspidlist_t structure.

asashofpid (3X)

Returns the ASH associated with a specified PID.

setash (2)

Changes the ASH of the calling process.

The asallocash() function is like the command ainfo newash (see “About Array Session Handles (ASH)”). Only a program with root privilege can use the setash() system function to change the ASH of the current process. Unprivileged processes can create new ASH values but cannot change their ASH.

The functions summarized in Table 7-17 are used to enumerate the active ASH values at a specified node. In each case, the list of ASH values is returned in an asashlist_t structure.

Table 7-17. Functions for ASH Interrogation



aslistashs (3X)

Returns active ASH values from one node or all nodes of a specified Array via a specified server

aslistashs_array (3X)

Returns active ASH values from an Array by name

aslistashs_server (3X)

Returns active ASH values known to a specified server node

aslistashs_local (3X)

Returns active ASH values in the local node

asashisglobal (3X)

Tests to see if an ASH is global

Executing an Array Command

The ascommand() function is the programmatic equivalent of the array command (see “Operation of Array Commands” and the array(1) man page). This command has many options and can be used to execute commands in three distinct modes.

The command to be executed must be prepared in an ascmdreq_t structure, which contains the following fields:

typedef struct ascmdreq {
   char *array;            /* Name of target array */
   int flags;              /* Option flags */
   int numargs;            /* Number of arguments */
   char **args;            /* Cmd arguments (ala argv) */
   int ioflags;            /* I/O flags for interactive commands */
   char rsrvd[100];        /* reserved for expansion: init to 0's */
} ascmdreq_t;

Your program must prepare this structure in order to execute a command. The option flags allow for the same controls as the command line options of array.

The result of the command is returned as an ascmdrsltlist_t structure, which is a vector of ascmdrslt_t structures, one for each node at which the command was executed. Each ascmdrslt_t contains the following fields:

typedef struct ascmdrslt {
    char     *machine;  /* Name of responding machine */
    ash_t    ash;       /* ASH of running command */
    int      flags;     /* Result flags */
    aserror_t error;    /* Error code for this command */
    int      status;    /* Exit status */
    char     *outfile;  /* Name of output file */
    int      ioflags;   /* I/O connections (see ascmdreq_t) */
    int      stdinfd;   /* File descriptor for command's stdin */
    int      stdoutfd;  /* File descriptor for command's stdout */
    int      stderrfd;  /* File descriptor for command's stderr */
    int      signalfd;  /* File descriptor for sending signals */
} ascmdrslt_t;

The fields machine, ash, flags, error, and status reflect the result of the command execution in that machine. The other fields depend on the mode of execution.

Normal Batch Execution

To execute a command in the normal way, waiting for it to complete and collecting its output, you do not set either ASCMDREQ_NOWAIT or ASCMDREQ_INTERACTIVE in the command option flags.

Control returns from ascommand() when the command is complete on all nodes. If the ASCMDREQ_OUTPUT flag was specified, and if the command definition does not specify a MERGE subentry (see “Summary of Command Definition Syntax”), the outfile result field contains the name of a temporary file containing one node's output stream.

When the command is implemented with a MERGE subentry, there is only one output file no matter how many nodes are invoked. In this case, the returned list contains only one ascmdrslt_t structure. It contains the ASCMDRSLT_MERGED and ASCMDREQ_OUTPUT flags, and the outfile result field contains the name of a temporary file containing the merged output.

Immediate Execution

When a command has no useful output and should execute concurrently with the calling program, you specify the ASCMDREQ_NOWAIT option. In this case, output cannot be collected because no program will be waiting to use it. Control returns as soon as the command has been distributed. The result structures do not reflect the command's result but only the result of trying to start it.

Interactive Execution

You can start a command in such a way that your program has direct interaction with the input and output streams of the command process in every node. When you do this, your program can supply input and inspect output in near real time.

To establish interactive execution, specify ASCMDREQ_INTERACTIVE in the command option flag. Also set one or more of the following flags in the ioflags field:


Requests a socket attached to the command's stdin.


Requests a socket attached to the command's stdout.


Requests a socket attached to the command's stderr.


Requests a socket that can be used to deliver signals.

As with ASCMDREQ_NOWAIT, control returns as soon as the command has been distributed. Each result structure contains file descriptors for the requested sockets for the command process in that node.

Your program writes data into the stdinfd file descriptor of one node in order to send data to the stdin stream in that node. Your program reads data from the stdoutfd file descriptor to read one node's output stream.

You will typically use either the select() or the poll() system function to learn when one of the sockets is ready for use. You may choose to start one or more subprocesses using fork() to handle I/O to the sockets of each node (see the select(2) , poll(2), and sproc(2) man pages). (You may also use sproc() to make subprocesses, but keep in mind that the libarray is not thread-safe, so it should only be used from one process in a share group.)

Executing a User Command

The asrcmd() function allows a program to initiate any user command string on a specified node. This provides a powerful facility for remote execution that does not require root privilege, as the standard rcmd() function does (compare the asrcmd(3) and rcmd(3) man pages).

The asrcmd() function takes arguments specifying:

The returned value (as with rcmd()) is a socket that represents the standard input and output streams of the executing command. Optionally, a separate socket for the standard error stream can be obtained.

Secure Array Services

This section provides more detailed information about Secure Array Services (SAS) and covers the following topics:

Differences between Standard and Secure Array Services

This section describes the differences between standard Array Services (arraysvcs) and Secure Array Services (sarraysvcs). Secure Array Services is built with OpenSSL version 0.9.7. It uses Secure Sockets Layer (SSL) to communicate between arrayd daemons.

Note: It is possible to relink the Secure Array Services (SAS) library /usr/lib/libarray.a with another version of OpenSSL to generate a new /usr/lib/ library. Since the resulting library is not tested by SGI, only limited support is provided for this kind of re-configuration. See /usr/lib/array/README.libarray for further instructions.

You need to make sure to properly protect access to various certificate files since they contain private information and keys for accessing the software.

A summary of the differences between standard Array Services and Secure Array Services is, as follows:

  • You cannot install and run standard Array Services (arraysvcs) and Secure Array Services (sarraysvcs) on the same system.

  • All the hosts in an array must run either arraysvcs or sarraysvcs. The two versions cannot operate at the same time.

  • The daemon for Secure Array Services is sarrayd . For standard Array Services, it is arrayd.

  • Secure Array Services requires secure shell (SSH) to be installed.

  • The chkconfig flag is, as follows:

    • array for standard Array Services

    • sarray on Secure Array Services

  • The startup script is, as follows:

    • /etc/init.d/sarray for Secure Array Services

    • /etc/init.d/sarray for standard Array Services

  • The arshell(1) command is not available for Secure Array Services.

  • For Secure Array Services, all command requests are sent to the local sarrayd running on the current host. This is for security reasons.

  • Secure Array Services requires OpenSSL images to be installed in order to run.

  • On Secure Array Services, the AUTHENTICATION parameter is set and enforced to NONE since certificates are used from the server side and the client side.

  • On Secure Array Services, some additional arrayd.conf parameters are available, as follows:


    Default setting





  • The default certificate installed in /usr/lib/array/cert requires a local user ( normally arraysvcs ) to have group read access ( normally arraysvcs ) to the files in this directory. This means any user defined for a particular command section must have the same group access.

    On Secure Array Services, is important to make sure group-read access is restricted to very few accounts. Not doing so can compromise the security features of SAS.

Secure Array Services Certificates

Certificates are used for authentication and for negotiating encryption of subsequent traffic between the sarrayd daemons in an array. The current implementation require the server and client certificates to be present. Upon starting, the /etc/init.d/sarray script attempts to create the required certificates using the makecert script. Certificates are not over-written. The default certificate, upon installation, allows a host to run stand-alone.

Note: The first invocation of the sarray services may take from five to ten minutes because it has to generate the Diffie-Hellman keys required for proper certificate exchange.

If it is required to run Array Services in a cluster, you need to sign SAS root certificate common to the entire cluster (see the gencert command information that follows later in this section). The SAS root certificate can be self-signed (default) or signed with any valid certificate obtained from an external source.

The layout of certificate files is, as follows:

  • /usr/lib/array/cert directory; characteristics are, as follows: permissions=750, owner=root, group=arraysvcs or the group defined in the arrayd.conf file. It contains the certificate and the Diffie-Hellman keys.




    Array Services root certificate. Self- signed or signed by an external source ( see the makecert -k option )

    client.pem and server.pem 

    Client and server certificate signed by root.pem


    Diffie-Hellman keys for certificate exchange

  • /usr/lib/array/cert/keys directory; characteristics are, as follows: permissions 750, owner=root, group= arraysvcs or the group defined in arrayd.conf. It contains the passphrase file leading to the private keys. Note passphrase are randomly generated.

  • If the group is different than arraysvcs or has been changed, use makecert -X to perform the required adjustments. See makecert -h for help. Note this will adjust keys and certificate ownership and permissions according to arrayd.conf user and group in local section. If not defined, arraysvcs is used for group and user. On IRIX, passwd.add script is used to add/update entries in /etc/passwd and /etc/group.

  • You can generate certificate for an entire cluster by running the gencert command. See gencert -h for help.

There are two certificate utilities available. They both use /usr/bin/openssl command-line utilities. They reside in the /usr/lib/array directory and can only be executed by root. Descriptions of these certificate utilities are, as follows:

  • The makecert utility is used to manipulate certificates for Secure Array Services. You can use the makecert -h command to obtain more information. A portion of the help output is, as follows:

    makecert -C {root|serverCA|server|client|dh} ...
                 makecert -K
                 makecert -v certificate_file_name options
                 makecert { -i|-I } {root|serverCA|server|client|dh}
                 makecert { -g|-G } [ -P password_length ] \
                                     [ -D duration_in_months ] \
                                     [ -k signing_key_file ] \
                                     [ -S subject_prefix ] \
                                     [ -H subject_FQHN ] \
                               {root|serverCA|server|client|dh} \
                               [ signing_certificate_file ] \
                               additional_certificates_to_be_included ]
                 makecert -X {new_group_ownership for certificate}
                 where:-C    Clean directory specified on command line
                       -K    WARNING: perform a
                             rm -rf /usr/lib/array/cert/
                       -i    Install/do_not_overwrite command-line
                             files under /usr/lib/array/cert directory.
                       -I    Same as -i but overwrite files.
                       -g    Generate certificate or Diffie-Hellman
                             keys using /dev/urandom only.
                       -G    Same as -g but not limited to /dev/urandom

  • The gencert utility is used to generate certificates for multiple hosts, generally for an array defined in the arrayd.auth file. The first host on the command-line will serve as the root certificate. It uses makecert to generate certificates. You can use the gencert -h command to obtain more information. A portion of the help output is, as follows:

    gencert [ -hdC ] [ hostnameRoot hostname ] ...
                 gencert [ -hdC ] -k signCertFile,signKeyFile
                                          hostname hostname ...
                 where: -h    this help message
                        -d    Also generate Diffie-Hellman 1024
                              bits keys
                        -C    Do not clean non-local entries after

    The gencert command generates all the necessary certificates for the specified hostname on the command line. For the current host, certificates are installed in /usr/lib/array/cert. For the remaining hosts, a tar file is created for each one.

    The first utility expects the first hostname to be the base root for the remaining hosts. If no hosts are supplied, the current host is used.

    The second utility is used when an external passphrase (the one that generated the signing private key) is used. In this case, root.pem is signed by the specified keys.

In IRIX, there is only one utility to add/update the user in /etc/passwd and /etc/group. The utility resides in the /usr/lib/array directory and can only be executed by root. A portion of the help output for passwd.add -h command is, as follows:

passwd.add -u {uname},{uid},{uid_max} \
                        -g {gname},{gid},{gid_max}
where: -u    Specify username,uid,uid_max
       -g    Specify groupname,gid,gid_max

The passwd.add command adds the specified username and groupname to the /etc/passwd file. If /etc/passwd.sgi file exists, a "noshow" is added. If /etc/shadow file exists, a pwconv is executed. No actions are performed if the specified entries already exist.

Secure Array Services Parameters

Currently, there are two parameters for Secure Array Services in the arrayd.conf file, as follows:


Default = 1. This should be changed if SAS root certificate are not self-signed.



Secure Shell Considerations

Secure Array Services requires that some version of Secure Shell (SSH) has been installed and is running. Secure pArray Services has been tested with OpenSSH versions 3.9p1, but it should work with any version of SSH. The only known requirement for the current version of Secure Array Services is that the SSH implementation installed support the following options:




Requests SSH to go to background just before command execution.

-i identity_file 

Selects a file from which the identity (private key) for RSA or DSA authentication is read.

Any authentication strategy supported by SSH can be used for Secure Array Services. However, having to enter passwords for every host where application processes will execute is tedious and prone to error, as a result SGI discourages the use of such an approach for authentication. Also, executions of MPI applications launched via batch schedulers cannot be authenticated interactively. A better approach to authentication via SSH is the use of key agents, such as ssh-agent, or an unencrypted key. For additional information about these approaches we recommend you consult SSH, The Secure Shell: The Definitive Guide from O'Reilly publishing.

Since there is no standard defined location for the ssh client command, the full path for the desired client can be specified using the ARRAY_SSH_PATH environment variable. If this environment variable is undefined it is assumed that the client command is named ssh and resides in the defined PATH of the user.

SSH allows users to authenticate using multiple identities. Support for this is provided in Secure Array Services via the ARRAY_SSH_IDENT environment variable. If this environment variable is set the value will be used as the identity for authentication when the ssh client is authenticating on the remote system. The use of identities is particularly useful when different authentication methods depending upon if the user is trying to authenticate for an interactive session or a batch session. If this variable is undefined, the default identity of the user is used.