Failure recovery for the NLB is possible by specifying a list of nodes for the NLB_SERVER variable in the nqeinfo(5) file. The collectors will update the NLB database on each specified NLB server so that all nodes have up-to-date information. The NLB clients query the NLB servers one by one in the configured order until they find one that is running or until they exhaust the list. The one potential problem with NLB failure recovery involves using the cevent(1) command for Job Dependency. For more information, see “Multiple NLB Servers and Job Dependency” in Chapter 12.

There is no failure recovery for the NQEDB node. If the NQEDB node goes down or loses network access, the cluster becomes a set of individual NQS servers. If jobs finish while the NQEDB node is inaccessible, the NQE database will not be updated and the jobs will still be marked as running in the NQE database.

The NQE database is limited to 36 connections. Four permanent connections are used by the NQEDB sever and one permanent connection is used by each lightweight server. Client connections are all transient and therefore limited only by performance considerations. However, if all 36 connections are used by servers, there is no connection left for clients connections. Therefore, the maximum number of nodes that may be in an NQE cluster is somewhere between 30 and 35, depending on the amount of expected client traffic.

Cluster rerun of jobs is not available with NLB scheduling.

Cluster rerun of jobs is available with the NQE database. However, if there are network problems between the NQEDB sever and an NQS server, the NQS server will appear to be down. In this situation, the NQEDB sever will rerun the job on another server; this results in the job running twice.

Performance is better with the NLB. The NQE database and scheduler are written in TCL, an interpretative language, and is slower. Also, you can run the NLB without the NQEDB sever, but you can not run the NQEDB sever without the NLB, if you are interested in total cycles.

NQS queues give users a rough idea of the relative position of their jobs on that node.

In the NQE database implementation, there is no way for users to understand their job request priority relative to other job requests in queued state.

The NLB allows sites to define policies for destination selection (routing) but not for priority scheduling of job requests. Priority scheduling is strictly first in, first out (FIFO).

The NQE Database as shipped uses FIFO scheduling but also allows sites to define their own schedulers using TCL scripts. This flexibility adds a level of complexity, as well as the obvious level of effort. However, the following types of scheduling would be possible with a customized NQE Scheduler:

The NQE database server (mSQL) serves connections from clients in the cluster or locally to access data in the database. The database used is mSQL, and it has the following features:

The monitor system task is responsible for monitoring the state of the database and which NQE database components are connected. Specifically, it determines which system tasks are connected and alive and, optionally, it purges old requests (called tasks when they are within the database) from the database.

A system task is an application that remains constantly connected to the database and performs a special task.

The scheduler system task analyzes data in the database, making scheduling decisions. Specifically, it does the following:

Administrator-defined functions may be written in Tcl to perform site-specific scheduling (see Chapter 10, “Writing an NQE Scheduler”).

The lightweight server (LWS) system task is a bridge to NQS, which performs the following tasks:

The following client commands give access to the NQE database; for additional information, see the man page for the specific command or see the NQE User's Guide, publication SG-2148:

The NQE database is comprised of objects. Each object is, in turn, made up of attributes, which are simply name and value pairs. An instance of an object in the NQE database can contain any number of attributes.

Each instance of an object always contains at least the following attributes:

The following object types are defined:

The task owner is the system task that is currently responsible for a given user task. Only the task owner may modify the attributes of a task. The task owner is defined in the attributes t.sysclass and t.sysname.

The name of a system task is expressed in the following form:

class.name

For example, the default scheduler's name is scheduler.main.

Each system task regularly queries the NQE database to find any new user tasks it owns.

The scheduler chooses a host on which the user task will run and then changes ownership of the user task to the LWS system task for that host. The LWS system task then submits the task to the LWS daemon on that host. The LWS daemon issues the job request to the local NQS system.

User tasks have a state attribute (t.state). Different states signify different stages in the progress of the task through the system. There is no limit to the number of states a user task may pass through, and the name of a state is arbitrary. However, there are some well-defined states.

A task's state may be changed without changing the owner. Also, the owner of a task may be changed without changing its state.

Figure 9-3 shows some ownership changes and state changes possible for a user task.

Figure 9-3. User Task Owners and States

A typical path for a user task is as follows:

Events form the mechanism for communication among users, administrators, user tasks, and system tasks.

Any client program connected to the NQE database may insert and post an event by inserting an object of type e. System tasks regularly check for any new user tasks they own. Once an event has been acted upon, it is marked as acknowledged and may be removed from the NQE database. (The monitor system task can be configured to automatically purge old events from the NQE database.)

Users may only post the signal event (by using either NQE GUI menu selection or the cqdel command) when it is targeted at user tasks they submitted. System tasks and administrators may send events to any object (including system tasks).

To send an event, use the nqedbmgr event post or ask command. For example, to post the event xyz with value 123 to object s23, use the following command; case is ignored for xyz in the example:

event post s23 xyz 123

Table 9-1 lists the standard events:

Every client user (including a system task) wishing to connect to the mSQL daemon requires a valid database user entry in a file called nqedbusers; the nqedbusers file resides in the NQE spool directory defined by NQE_SPOOL in the nqeinfo file. When connected, any object (that is, request) inserted into the database is owned by the dbuser. The dbuser name may be a string of any alphanumeric characters and does not necessarily have to be a UNIX user name (dbusers do not need /etc/passwd entries on the machine running the mSQL daemon).

The mSQL daemon reads and uses the contents of the nqedbusers file to determine whether a given client may connect to the database. If, as a result of this check, the client user's connection request is refused, a Connection disallowed message is passed back to the client and the connection is broken.

The nqedbusers file is made up of an optional %host segment followed by multiple %dbuser segments. Blank lines and lines beginning with # are ignored.

The host segment specifies an optional list of client host names explicitly allowed or disallowed access to the database. If the segment is not supplied, or the list of host names is empty, any client host may connect to the database, and the checks move on to the dbuser segments. The syntax in the nqedbusers file is as follows:

%host
      [clienthostspec
      clienthostspec
      ...]

A clienthostspec is a host name optionally prefixed by + (to allow access explicitly) or -- (to disallow access).

The database user segment defines allowed or disallowed access for a specified database user. The syntax in the nqedbusers file is as follows:

%dbuser dbusernamespec privtype
       [clienthostspec clientuserspec
       clienthostspec clientuserspec
       ...]

The dbusernamespec is the dbuser name and may be optionally prefixed by -- to explicitly disallow the dbuser. The following formats allow any user to connect to the NQE database:

%dbuser
%dbuser +
%dbuser + user

If the dbusernamespec is not supplied, or the list of dbuser names is empty, any user may connect to the database. A dbusernamespec of + alone has a special meaning and corresponds to a dbuser equal to the client user name. This entry allows access for many users using their existing UNIX user names without the necessity of explicitly specifying each name in the file. The %dbuser + user format allows any user to connect to the NQE database; user is the privtype.

System tasks (monitor, scheduler and LWS) connect using a dbuser of root.

privtype specifies the type of connection privilege. Two types are defined, as follows:

clientuserspec is a UNIX user name optionally prefixed by + (to allow access explicitly) or - (to disallow access). An entry of simply + allows any client to connect as the given dbusername.

An example of an nqedbusers file follows:

# nqedbusers file
#
# Allow root access from any host
%host
       # Disallow hosts
       -badhost
       # Allow other hosts
       +
%dbuser root system
       localhost root
       server1 root
       server2 root
%dbuser fred user
       client1 shirley
#
# Allow any user access from any host.
# ( db username = client username )
%dbuser + user
       -badclient bob
       - amy
       + +

In this file, access is completely denied for host badhost, but all other hosts are allowed access. A dbuser of root has been defined with full access privileges (system) for hosts localhost, server1, and server2, which allows the LWS to run from those three hosts. A dbuser of fred has been defined and may be used by shirley from client host client1. All other users are allowed to use their UNIX names to connect as the dbuser name, with the exceptions of bob from badclient (bob can connect from anywhere except from badclient) and amy, who cannot connect from any host.

The administrator may define how the LWS validates jobs from the NQE database before submitting them to NQS locally. Valid values are as follows:

The initial value when NQE is first started is FILE. After that, you must update the global configuration object in the NQE database and post a TINIT event to the lws to tell it to re-read the configuration. For example:

% select s
Connected to coffee as DB user root (conn=3)

     ID      CLASS NAME              STATE
     s2    monitor main            Running
     s3  scheduler main            Running
     s4        lws coffee          Running
     s5        lws latte           Running
     s6        lws cappuchino      Running

% select -f s4 {run_mechanism}
Object s4:
         run_mechanism = File

% update s4 {run_mechanism Passwd}
Object s4 updated

% select -f s4 {run_mechanism}
Object s4:
         run_mechanism = Passwd

% event post s4 TINIT
Posted event e16516

Event objects are used to communicate between tasks, both system tasks and user tasks. For example, if you change the global configuration, you need to send a TINIT event to the appropriate system tasks in order to tell these tasks to re-read the configuration and get the new values.

Table 10-4, lists the predefined events you may want to send.

A dbuser connected with the privilege type of user may only send events to job tasks created by the same dbuser. Those connected with the privilege type of system may send events to any task, including system tasks.

A dbuser connected with system privilege may read any object in the database by using the nqedbmgr interface. That is, any task may be examined.

A dbuser connected with user privilege is restricted to using the cqsub, cqstatl, cqdel, and NQE GUI interfaces.

The administrator may allow the user to see summary information on all tasks in the NQE database, not just the user's requests and tasks. This is achieved with the global configuration attribute status.access.

---

Note:: You do not need to post an event when you issue the following nqedbmgr commands.

---

The following nqedbmgr command allows users to see summary information on all requests and tasks (the default is all):

update config {status.access all}

The following nqedbmgr command allows a user to see only the requests and tasks the user created:

update config {status.access own}

Table 9-5 lists the predefined global configuration attributes for the NQE database (this table is also provided on the nqedbmgr(8) man page):

The default scheduler file is situated in the /nqebase/bin directory and is called local_sched.tcl. The default scheduler has the following properties:

The default scheduler can interpret the following predefined attributes, which are overridden if they are set in the system scheduler object. The following example sets the scheduler passrate attribute, which overrides the default value of 60:

nqedbmgr update scheduler {passrate 70}

Table 9-6 lists these attributes (this table is also provided on the nqedbmgr(8) man page).

Table 9-7 lists the predefined LWS attributes (this table is also provided on the nqedbmgr(8) man page):