After receiving a request in a load-balanced pipe queue, NQS will send a query to the NLB server. This query includes the name of the policy to be used.

The policy name is used to find the correct policy to apply. If the name is not that of a known policy, the default policy is applied. The default policy is the one named NQS.

NQS is shipped with a policy named nqs, which selects batch queues based on which system has the highest average idle CPU load.

The first constraint is taken in and applied to the list of hosts that have sent data to the server by using the ccollect process (described in “Starting NLB Collectors” in Chapter 7). It is applied to the objects of object type NLB. For a list of these objects, use the following command:

nlbconfig -olist -t NLB

A constraint is expected to be a Boolean expression, and a true result means that the host is kept as a valid target. A false result means the host is discarded.

Each subsequent constraint is applied to the hosts that met the previous constraints. This process continues until no more constraints are left, or until no hosts meet a constraint. If there are no acceptable hosts, NLB returns a null list, and NQS retries the request later. If there are no constraints, all hosts are assumed to be acceptable targets for the policy.

Constraints can filter out machines that physically cannot run a request, as in the following example:

# Policy for SPARC-specific requests
policy: SPARC
constraint: [ NLB_HARDWARE == "SPARC" ]

For the nqs policy, any machines that do not have a queue name associated with them are filtered out. It is also important to filter out machines that have not sent data recently. The time period used to evaluate this depends on the time interval between samples sent from the collectors. The following example sets the time period to 300 seconds:

constraint: exists(NLB_QUEUE_NAME) && age < 300

Constraints can also remove machines that are currently not good choices. For example, the following policy selects only machines that have more than 32768 Kbytes of free memory (32 Mbytes). Memory is measured in units of Kbytes:

policy: bigjob
constraint: NLB_FREEMEM > 32768

After the list of hosts has been filtered, the sort expression/operation is used to order them. The expression/operation is applied to each host in turn, and the result for that host is stored in the NLB database as the NLB_POLICY attribute. The hosts are then sorted on this number. The host with the largest NLB_POLICY value is regarded as the best choice.

A policy can optionally specify a count of the maximum number of targets that should be returned. This is needed in large networks to reduce the number of different systems to which NQS might attempt to send a batch request.

Policies are based on statistical data gathered from machines by NLB collectors. This data reflects the state of the machine in the recent past, but its current state may be different.

The data available to the server is a dynamic snapshot of the machine state; therefore, the server's picture of the state of machines in the network is always a little out-of-date.

This section describes a special collection of attributes, which, by convention, are named NLB_A_regular attribute name, such as NLB_A_IDLECPU.

Because an instantaneous snapshot of machine load could cause an incorrect view of its true state over time, the collectors provide two sets of measurements: the instantaneous data (which is of most use in the NQE GUI Load window displays) and averaged data for use in policies. The averaged data tends to smooth out the short-term peaks and troughs in the instantaneous data and to give a more accurate view of the overall load, as can be seen in Figure 8-1 and Figure 8-2.

Figure 8-1. Average Machine Load Data

Figure 8-2. Instantaneous Machine Load Data

The averaged data (stored in attributes beginning with NLB_A_) are calculated by adding the instantaneous measures taken over the last n seconds and dividing by the number of samples taken (the ccollect -r option controls n).

You can tune the averaging of data in the policy to reflect the type of work being run on a machine. For example, you can user longer sampling periods on machines at which long requests are run. However, the longer the period you use, the longer it takes the measures to catch up with reality; for example, even after a large requests is finished, it takes time for the load it produced to work its way out of the data.

A network usually will contain a mix of machines: different types of machines and perhaps machines of the same type but differing capacity (such as CRAY T90 and CRAY J90 systems).

The measures sent by the collectors reflect what is actually happening on a particular machine, but they do not indicate how that activity compares to the maximum capacity of that machine, or how that machine compares to others in the network. To measure these factors, you must introduce weighting or normalization factors into policies.

For example, if you are concerned with the amount of idle CPU time available on a machine (which is reported as a percentage), you could write a policy that sorts on this factor, as follows:

policy: idlecpu
sort: NLB_IDLECPU

In this case, a small workstation may be selected as the best target for work in preference to a large Cray Research system, even though 10% idle time available on the Cray Research system could well represent much more computing power than 50% idle time on the workstation.

By adding weighting factors to machines, you can balance out this difference. If you define a new attribute called NLB_SITE_CPUNORM for each host (as in the example in “Example of Writing a Load-balancing Policy”), and assign values representing the relative CPU powers of the different machines (machine A is 100 while machine B is 10), you can factor this into the policy, as follows:

policy: idlecpu
sort: NLB_IDLECPU * NLB_SITE_CPUNORM

Of course, the relative powers of different machines are related to the nature of the request to be executed. A highly vectorized application probably requires a larger difference in weightings between a workstation and a Cray Research system than would a scalar application.

You may want to apply different policies at different times of the day or the week. For instance, you may want to allow requests to run on file servers at night when there is little NFS activity. Although there is no direct support for this in the NLB, you could use a cron job to load a new set of policies into the server at defined times. This job would replace the policies file and cause the server to reread it, as in the following example:

cp policies policies.day
cp policies.night policies
nlbconfig -pol

The following example would switch the policies files back again:

cp policies policies.night
cp policies.day policies
nlbconfig -pol

For this example to work, root must be in the master access control list (ACL) for the server defined in its configuration file, because this ACL controls who can update the policies file.

The following example selects all hosts that have the attribute NLB_QUEUE_NAME and for which the data has been updated within the last minute (AGE < 60). The hosts are then sorted by weighted idle CPU time (as described in “Other Factors: Weighting Machines”).

policy:       AGE
constraint:   exists(NLB_QUEUE_NAME) && AGE < 60
sort:         NLB_A_IDLECPU * NLB_SITE_CPUNORM

The following example selects all machines with an average system load of 10%/CPU (or less); NLB_A_SYSCPU is the average system CPU (percentage) and NLB_NCPUS is the number of CPUs:

policy:        default
constraint: [NLB_HARDWARE == "CRAY"] =>
               (NLB_A_SYSCPU / NLB_NCPUS) < 10.0
constraint: [NLB_HARDWARE == "sun"] => NLB_PHYSMEM >= 32768
sort:          NLB_NCPUS

The following example defines a new attribute used to normalize CPU loads. This is useful for a site with machines that have widely different CPU power. First a new object attribute is created, then values are assigned to the attribute, and finally the policy is added to the server NLB database so that data associated with it can be stored. All this can occur without any part of the system being restarted.

The NLB server supports mappings between the internal values used to represent objects and attributes, and the human-readable names and descriptions. The mapping is defined by configuration data held within the server. This data can be downloaded from the server and converted to a text form by nlbconfig. The nlbconfig program also can take the text form of the name map and load it back into the server, allowing you to add new definitions to the map. This text form of the name map is known as a name_map file.

The name_map file contains a sequence of definitions for different object types and their associated attributes. Each object has a name and an ID. Each attribute of an object has a name, a text description, and an ID formed from a type and a number. The attribute ID and name must be unique from all other attributes of that object.

A name map for an object of object type NLB (information about an individual host) would be as follows; column one is the attribute name, column two is the attribute description, and column three is the attribute type ID:

map NLB = 1024
{
         NLB_OSNAME              "OS name"                       string(1);
         NLB_RELEASE             "Release"                       string(2);
         NLB_VERSION             "Version"                       string(3);
         NLB_HARDWARE            "Hardware"                      string(4);
         NLB_QUEUE_NAME          "Queue name"                    string(5);
         NLB_TMPNAME             "Temp file path"                string(6);
         NLB_BSUSPEND            "Auto suspend state"            string(7);
         NLB_IDLECPU             "Idle cpu"                      float(1);
         NLB_RUNQLEN             "Run queue"                     float(2);
         NLB_SWAPPING            "Swap activity"                 float(3);
         NLB_POLICY              "Policy Weighting"              float(4);
         NLB_SYSCPU              "System cpu"                    float(5);
         NLB_RUNQOCC             "Run queue occupancy"           float(6);
         NLB_A_IDLECPU           "Ave Idle cpu"                  float(11);
         NLB_A_RUNQLEN           "Ave Run queue"                 float(12);
         NLB_A_SWAPPING          "Ave Swap activity"             float(13);
         NLB_A_SYSCPU            "Ave System cpu"                float(14);
         NLB_PHYSMEM             "Memory"                        integer(1);
         NLB_FREEMEM             "Free memory"                   integer(2);
         NLB_CLOCK               "Clock speed"                   integer(3);
         NLB_NCPUS               "Num CPUs"                      integer(4);
         NLB_PSWCH               "Context switches"              integer(5);
         NLB_NUMPROC             "Process count"                 integer(6);
         NLB_FREETMP             "Free temp space"               integer(7);
         NLB_KEYIDLE             "Keyboard idle time"            integer(8);
         NLB_SWAPSIZE            "Swap Device size"              integer(9);
         NLB_SWAPFREE            "Free Swap space"               integer(10);
         NLB_A_FREEMEM           "Ave Free memory"               integer(11);
         NLB_A_PSWCH             "Ave Context switches"          integer(12);
         NLB_A_NUMPROC           "Ave Process count"             integer(13);
         NLB_A_FREETMP           "Ave Free temp space"           integer(14);
         NLB_A_SWAPFREE          "Ave Free Swap space"           integer(15);
         NLB_IOTOTAL             "Total user I/O rate"           integer(16);
         NLB_A_IOTOTAL           "Ave Total user I/O Rate"       integer(17);
         NLB_IOREQS              "I/O Requests per sec"          integer(18);
         NLB_A_IOREQS            "Ave I/O Requests per sec"      integer(19);
         NLB_SYSCALLS            "System calls per second"       integer(20);
         NLB_A_SYSCALLS          "Ave system calls per second"   integer(21);
         NLB_TPDAEMONUP          "Tape Daemon running"           integer(22);

         NLB_MPP_UP              "Cray T3D available"            integer(23);
         NLB_NQS_UP              "NQS available flag"            integer(24);
         NLB_NQS_VALTYPE         "NQS validation type"           integer(25);
         NLB_TIMESTAMP           "Update time"                   time(4096);
         NLB_HOST                "Originating host"              string(4098);
}

You can change the attribute names (such as NLB_OSNAME) and the attribute descriptions. However, you cannot change the attribute type ID. The type ID numbers are built into ccollect(8); because of this, specific attributes always represent the same piece of information.

---

Note:: It is recommended that you do not change attribute names because they are used in X resource definitions for the NQE GUI Load window displays and in policy definitions.

---

Table 8-1 lists the allowed attribute types:

You can add new attributes and store values to the NLB database at any time by using the nlbconfig command. However, the data cannot be used anywhere until the attribute has been defined in the name_map file, even if objects in the NLB database possess the attribute. These site-defined values can then be used in policies and the X resource definitions for the machine load displays, which are displayed by using the NQE GUI Load window. For an example of writing a load-balancing policy, see “Example of Writing a Load-balancing Policy”.

A number of object and attribute names are reserved. They are hardcoded into the NLB collectors. A default name mapping file is supplied for these objects and attributes. Attribute numbers below 8192 are reserved for Cray Research. Table 8-2 lists the reserved host object attributes.

There are averaged versions of all integer and floating-point attributes except for NLB_POLICY, NLB_KEYIDLE, NLB_PHYSMEM, NLB_CLOCK, NLB_NCPUS, and NLB_SWAPSIZE. These attributes use the naming convention _A inserted in the middle of the name (for example, NLB_A_IDLECPU is the average version of NLB_IDLECPU).

Table 8-3 lists the availability of the attribute data across platforms. The following conventions are used in the table:

The NLB_TPDAEMONUP and NLB_MPP_UP attributes are collected only from UNICOS and UNICOS/mk systems.

The NLB also stores attributes for running requests. There is an attribute for each field in the cqstatl or qstat -f display (over 200 attributes are included). To obtain a list of these attributes, use the following command:

nlbconfig -mdump -t NJS | more

The output from this command lists attribute names, descriptions, and types. An abbreviated example follows:

# nlbconfig -mdump -t NJS |more
###############################################
# creation date: Wed Mar 22 15:08:13 1995
# created by: ljgm
# object timestamp: Sat Mar 18 11:56:52 1995
########
map NJS = 500
{
NJS_NQSID                  "NQS identifier"                string(1);
NJS_JOBSTAT                "Job status"                    string(2);
NJS_NUMPROC                "Processes in job"              integer(3);
NJS_RUNUSER                "Running user"                  string(4);
NJS_QUENAM                 "Queue name"                    string(5);
NJS_CPUREQLIM              "Per-request CPU limit"         integer(6);
NJS_CPUREQUSE              "Per-request CPU use"           integer(7);
NJS_MEMREQLIM              "Per-request memory limit"      float(8);
NJS_MEMREQUSE              "Per-request memory use"        float(9);
NJS_PFSREQLIM              "Per-request PFS limit"         float(10);
NJS_PFSREQUSE              "Per-request PFS use"           float(11);
NJS_ORGHOST                "Originating host"              string(12);
NJS_ORIGUSER               "Originating user"              string(13);
NJS_QUEENT                 "Position in queue"             integer(14);...

An object data file may be used to assign values to attributes for each object. Objects can be represented in a text format. Functions and commands are provided so administrators can create their own objects and insert them into the NLB database.

An object data file consists of a sequence of objects. Each object has a type, a name, and a sequence of assignment statements that define attribute values. The type names for objects and names for attributes are taken from the name map in the NLB database. The following example defines two attributes on three different hosts:

object NLB (cloudy) {
    NLB_SITE_CPUNORM = 10;
    NLB_APPLICATIONS  = "UniChem dbase";
}

object NLB (gust) {
    NLB_SITE_CPUNORM = 12;
    NLB_APPLICATIONS = "UniChem TurboKiva";
}
object NLB (hot) {
    NLB_SITE_CPUNORM = 24;
    NLB_APPLICATIONS = "";
}

The legal values for an attribute depend on its type as defined in the name map, as follows:

First you would update the name_map file with the new attributes NLB_APPLICATIONS and NLB_SITE_CPUNORM and load it into the nlbserver. Note that because of how the server code works any attributes that are to be used in policies must be in either the NLB object or the NLB_JOB object. If any attribute is used in the policy file that is not in either of these two objects the policies file will not be read successfully by the nlbserver.

name_map file:

map NLB = 1024
{
NLB_APPLICATIONS
"Resident Applications"
string(999);
}

Second, the appropriate objects must have attributes set properly. For this example, these are set with the following object data file that is downloaded to the nlbserver with the following command:

nlbconfig -t nlb -oupdat filename

object NLB (gust) {
NLB_APPLICATIONS = "Unichem TurboKiva Nastran";
}
object NLB (wind) {
NLB_APPLICATIONS = "Nastran";
}
object NLB (rain) {
NLB_APPLICATIONS = "Unichem";
}
object NLB (frost) {
NLB_APPLICATIONS = "TurboKiva";
}

Third, write some policies that use the attribute in question.

Policy file:

policy:
first
constraint:
[NLB_APPLICATIONS == "Nastran"]
sort: (NLB_A_IDLECPU +5)/10000
policy:
second
constraint:
[NLB_APPLICATIONS == "Unichem"] ||\n  [NLB_APPLICATIONS == "TurboKiva"]
sort: (NLB_A_IDLECPU +5)/10000

Finally see the results of the policies when they are invoked.

Output from nlbpolicy:

nlbpolicy -p first
gust
0.0092525
wind
0.003213

nlbpolicy -p second
gust
0.009225
rain
0.004950
frost
0.000500

To configure an NQS destination selection environment, define a pipe queue as a destination selection pipe queue on each system that will route requests using NLB (as described in “Example of Load Balancing Using the Default Policy”). No destinations should be defined for this pipe queue. The queue must be marked as a destination selection pipe queue (CRI_DS). (For additional information, see “Commands for Configuring Destination-selection Queues” in Chapter 5.)

The collectors on your NQE nodes have been reporting their availability. When you submit a request to the new queue, the NLB selects one of these servers as the destination queue for your job.

If the qmgr default batch queue is set to a destination-selection pipe queue, all requests that do not designate a specific queue are routed using NLB. If the default queue is not a destination-selection queue, you can use the General Options Queue Name selection of the Configure menu on the NQE GUI Submit window, use the -q queues option, or designate the queue in the script file to select a destination-selection queue.

If you are using file validation, each user must create a .rhosts or .nqshosts file on each system to which a request might be routed by the NLB. If you are using password validation, the user's password must be the same on all systems to which the NLB could route the request.

The following example could be used as a policy definition in the NLB policies file:

# Policy for load-balancing between Cray systems
# Note that NLB_HARDWARE can be "CRAY J90", "CRAY Y-MP",
# "CRAY T3E", etc. and still match [ NLB_HARDWARE == "cray" ]
# ( AGE < 300 ) for systems that have sent data in the
#      last 5 minutes
# (NLB_PHYSMEM - NLB_A_FREEMEM....) for a ratio of memory used
# 	   (real and swap) to real memory of less than 4
# (NLB_NCPUS <= 2.... for less than 20% system CPU per CPU for
#      hosts with 1 or 2 CPUs
# (NLB_NCPUS > 2.... for less than 3.5% system CPU per CPU for
#      hosts with more than 2 CPUs
#
#Sorted by ratio of number of CPUs to length of the run queue
#
policy: RUNQUEUE
constraint:[NLB_HARDWARE == "cray"] && age < 300
constraint:(NLB_PHYSMEM - NLB_A_FREEMEM +
               (NLB_SWAPSIZE - NLB_SWAPFREE) * 1024)
                  /NLB_PHYSMEM < 4
constraint: NLB_NCPUS <= 2 => NLB_A_SYSCPU / NLB_NCPUS <20
constraint:(NLB_NCPUS > 2) => (NLB_A_SYSCPU /  NLB_NCPUS <3.5)
sort:        NLB_NCPUS / NLB_A_RUNQLEN

Destination selection can be used to limit requests when host resources reach a threshold. For example, the following policy omits any host with system CPU usage greater than 10%. This can keep a busy system from receiving more requests.

policy:      AGE
constraint: AGE < 300
constraint: NLB_A_SYSCPU < 10
sort:          NLB_A_IDLECPU * NLB_NCPUS

NQS destination-selection queues used for load balancing specify the policy for that queue. For this reason, you may want to have multiple load-balancing queues for requests with disparate resource requirements. For example, CPU-bound requests could be routed to a machine with idle CPU but little free memory. The following policies, MEMORY and CPU, serve this purpose. The first limits hosts to those with more than 32 Mbytes of memory and a low memory demand, calculated by the memory oversubscription. Hosts are then sorted so that larger memory machines are chosen first.

The CPU policy uses additional NLB attributes that would be added by the site. NLB_SYSCPU_LIMIT would be defined as an integer value that could be different for each host. NLB_RUNQLEN_LIMIT also could be different for each host. The CPU policy limits the system CPU time and the run queue length. Hosts are sorted so that those with idle CPU time are first, and those with more CPUs are preferred. If there is no idle CPU time, the hosts are sorted by the number of CPUs.

policy:      MEMORY
constraint: NLB_PHYSMEM > 32768
constraint: (NLB_PHYSMEM - NLB_A_FREEMEM +
               (NLB_SWAPSIZE - NLB_SWAPFREE) * 1024)
                 / NLB_PHYSMEM < 3
sort:          NLB_PHYSMEM

policy:       CPU
constraint: NLB_A_SYSCPU / NLB_NCPUS < NLB_SYSCPU_LIMIT
constraint: NLB_A_RUNQLEN / NLB_NCPUS < NLB_RUNQLEN_LIMIT
sort:         NLB_A_IDLECPU * NLB_NCPUS + NLB_NCPUS

This section discusses NQS behavior when processing a batch request submitted to a destination selection queue.

When a batch request is submitted to a destination selection queue, it obtains a list of destinations from the NLB server. The list of destinations is tried one at a time until the request is accepted by one of the destinations or until the list is exhausted. If no destination accepts the request, NQS waits and tries again, based on the interval set by the qmgr command set default destination_retry time (the default is 5 minutes). NQS continues to try to deliver the request to a destination until the request is accepted or the request has waited the maximum allowable time, which is defined by the qmgr command set default destination_retry wait (the default is 72 hours). If no destination accepts the request during the maximum time, the request is deleted and the submitting user is notified by mail.

One of the following situations also may occur:

Figure 8-3. NQS Configuration with Load Balancing by Default

The following qmgr commands create the pipe queue nlbqueue and define it as the default queue for batch job requests:

% qmgr
Qmgr: create pipe_queue nlbqueue server=(pipeclient CRI_DS) priority=33
Qmgr: set default batch_request queue nlbqueue
Qmgr: quit

All requests that use the default queue are routed to a destination host by the destination selection algorithm. The NLB policy (for load balancing) defaults to nqs. The NLB server host defaults to NLB_SERVER.

In this scenario, a site has multiple NQS systems. The systems are similar, but there is production work that must run on specific systems. There is also work that could run on any available system. The default is that requests are submitted to a standard pipe queue. Users are encouraged to choose a destination selection queue when they submit work that can run anywhere in the complex.

This situation can be accommodated by defining two pipe queues, as shown in Figure 8-4. One is a standard pipe queue, called standard. The second, called nlbqueue, is configured to perform destination selection.

Figure 8-4. NQS Configuration with Load Balancing by Selection

The following qmgr commands create these pipe queues and set the default queue to standard so that requests are routed by standard NQS routing. The pipe queue standard explicitly defines a policy name, NLB host, and NLB port.

Qmgr: create pipe_queue standard server=(pipeclient) \
destination=([email protected],[email protected]) priority=30
Qmgr: create pipe_queue nlbqueue \
server=(pipeclient CRI_DS CPU cool:nlb) priority=30
Qmgr: set default batch_request queue standard

Requests that are submitted to the queue nlbqueue are routed to the most lightly loaded system.

If you have three different application packages, NASTRAN, Oracle, and UniChem, which are used by requests but not available on all machines, you would set up a policy as described in this section. To let users specify on the command line which application packages are required, you can perform the following steps:

Such job_ attributes can be used for any purpose. They are not restricted to the presence or absence of applications on hosts.

You can test a policy by using the nlbpolicy -p option. Its syntax is as follows:

nlbpolicy -p policyname

Optionally, you can use the -a option to specify any attribute described in Table 8-4, or you can create your own attributes and add them to the name map. Valid attributes are defined in the NLB for object type NLB_JOB. They must begin with the characters JOB_ and can be either a decimal integer or a float. The value can be either a decimal or a float, in normal or scientific notation. You cannot specify negative, hexadecimal, or octal numbers.