One of the unique features of Events is its hardware inventory tracking mechanism. At system start-up, Events assembles an inventory record, including hard disks, tape drives, RAM, network interfaces, and software where applicable. If a list exists from a previous start-up, then the two lists are compared. Additions and deletions are reported via email and to the EMD.

On most systems, Events also includes a software inventory. The software inventory process is done similarly to the hardware inventory tracking mechanism. You can set how frequently the software inventory is generated in the testtab file. See “The testtab.<hostname> File” for more details.

Events communicates to SNMP management systems via SNMP. As an SNMP agent, Events initiates error messages and alerts, and provides information to the SNMP management system. By responding to inquiries from the SNMP management system, Events makes workstation monitoring an interactive process.

For the System Manager, Events is a process that runs in the background and looks after the system. The System Manager can specify whether or not measured values are stored to a database, who should be notified if a test fails, and how to notify someone when a test fails for each test that is performed. Also, other tests can easily be added to the built-in tests suite.

For the Network Manager, Events is an SNMP agent that emits enterprise-specific traps to notify the appropriate Network Manager of a failed test condition. All tests are also manageable via SNMP. MIB II (Management Interface Base) is also supported.

Each Events test measures some numeric value. It is easier to figure out which alarm threshold is the most appropriate if you know what the value represents. The following examples use tests that are the most often misunderstood.

The paradigm is:

This test looks in ASCII log files for the recent addition of message types you have defined. Though an alarm message may contain text from the monitored file, the threshold itself is not the message text; it's the number of matched string entries found.

Since the test value (count) is the number of matches found (regular expression matches), you must set one of the alarm thresholds if you desire an alarm. By setting the High Level limit equal to one, an alarm occurs every time an offending message is found. Only the first instance of any given threshold is reported.

Let's say, for example, that you want to look for the presence of a hardware failure by “scanning” the /var/adm/syslog. The value matches should be set to 1. An alarm will occur as soon as Events detects the presence of the desired message string.

In another instance you may not be concerned about a particular message per say, but rather the frequency of a particular message, such as the number of hardware error messages in /var/adm/syslog within a given period of time. Set the Threshold value to the number of messages to look for and the test frequency to the amount of time in which that number of messages constitutes an alarm condition.

This test checks the time stamp associated with the file. The time stamp is actually a number. If the number increases, the file has been accessed. The number should never decrease. A decrease in value would be suspicious and may indicate a security breach.

A value of 1 for the Pos Jump Threshold is sufficient to constitute a change in the timestamp being checked.

This test counts the number of processes containing the same name that are running.

Events are not sent for the first and last instance of a process. A low threshold value of zero tests for the absence of a process. Setting the low threshold value higher than the high threshold will disable the low threshold testing.

This test counts the number of processes running as specified by the name value used in testname and arguments field. Minimally, the testname must be the filename portion of the binary used as the process to test. Use the arguments field to further refine process matching.

This test looks at the size of processes you've specified. Some platforms provide identification of the process in long name (fully qualified pathname) as well as short name (process name) formats as specified in the Testname field of the Add or Modify Events Test window. Use the arguments field to further refine process matching.

AgentMon performs all tests. When an alarm condition occurs, AgentMon may notify someone via traditional methods, such as email specified by the local user, and/or it may also notify network management via an enterprise-specific SNMP trap.

AgentENL is an interface between the AgentMon SubAgent and the network management application, such as SunNet Manager or HP OpenView. AgentENL must be started before AgentMon starts, if you intend to use the SNMP interface.

You can also use EventsCli with any current monitoring scripts and programs to handle any special needs that AgentMon can't cover. By calling EventsCli from the alarm notification section of your script, you gain much greater flexibility and control over how the alarm is treated. EventsCli will send an SNMP trap, it will log the alarm to the EMD, and it will notify PEP of its activity. Refer to Appendix F, “Events Commands,” for more information.

Events complies with the following Internet standards:

Logging to the EMD (Enterprise Management Database) is enabled by specifying the parameter log for each test from which you wish to collect data. Also, you can specify a value for delta for each test you want to log. Logging will then occur if the previously logged value varies by more than delta units from the current value. Setting delta=0 results in the measured value always being logged.

Delta

You can use delta to set your tests so they monitor more frequently, log less often, and still not lose any logging information. This is most useful in one of two situations:

An example of the first case: Monitoring a file that seldom changes size

With traditional logging (log every measurement), if you monitor this file once per minute and its size changes only once each hour, the other 59 out of 60 loggings are identical. With delta logging, you can specify that logging should only occur if the value changes significantly.

In this case, you could specify you want logging to occur only if the file size changes by one or more bytes (set delta=1). Now the log only contains the same two critical points (and not the other 58 instances of the repeated measurement).

An example of the second case: Monitoring an object whose value has a greater precision than is required

Suppose you are monitoring a filesystem's free space and logging the data so you could later use the data to help predict when to buy additional storage. If you set delta=1000, then logging will only occur when the available disk space changes by >1000 units from the previously logged value. In effect, you are specifying the resolution of your data.

Alarm Messages

If the test generates an alarm message, logging will always occur (even if the test has logging disabled). AgentMon also logs itself each time it is started, restarted (warm start), or normally terminated.

You can use a command parameter, which is a pathname, in the testtab entry for a test to perform some automated corrective activity. The pathname is assumed to be the name of a user-provided program. When an alarm condition occurs, the named process is started with the following arguments (listed in order):

Examples of corrective action scripts can be found in the subdirectory $ENLIGHTEN/contrib.

You can also include the pep capability in the testtab entry to notify PEP of the alarm.

Each test, or testtab entry, is composed of a test name and optional parameters that serve as “keywords” to define the scope and behavior of the test. The parameters are:

The following is an example test entry using some of these parameters.

cpu load |:\
on:testfreq=1:alarmfreq=60:mailer=/bin/mail:notify=root:log:\
high=5.0:units=units:delta=50:pep:command=/policy/myproc:\
/* end cpu load */:

Refer to Appendix G, “Sample Events Files,” for a sample of a complete testtab file.

Threshold types used in the testtab.<hostname> file can be specified as absolute values, percentage change, or incremental change based on the unit of measure for the particular test. Any combination of threshold types can be set. AgentMon checks the current values for each test and compares the measured value against these thresholds and sends alarms based on the following order of precedence:

Most tests measure integer values; others measure floating point values. Ensure that the values supplied for threshold types are consistent with the Unit of Measure for the particular test being configured. Failure to do so, however, will only result in rounding errors.

The jump and rate thresholds compare the current test value with the last measured value (change over time). For process monitoring, the process built-in first/last alarm is checked first, then the alarm thresholds are checked in the order listed in the preceding table (see “Processes Tests” for more details).

The following table shows the CPU tests.

The following Kernel Group tests may be of occasional usefulness to your local UNIX performance expert and can also be an excellent troubleshooting aid for certain kinds of problems. These tests do not normally need to be active.

The following table shows the File System tests.

The following table shows the Printer test.

The following table shows the Process test.

The following table shows the Inventory tests.

The following table shows the Remote Procedure Call (RPC) tests. These tests report various types of errors that can occur with the RPC protocol. This information is not normally required, but can be very useful when tracing network problems related to RPCs. Alarm thresholds may be set for each test, but there are no TRAPs associated with them.

Your local network specialist and O/S provider can provide more specific information about these tests. Specifics vary from O/S to O/S and most O/Ss do not support everything on this list.

---

Note: The RPC statistics are not available on HP/UX.

---

The following table shows the Virtual Memory (VM) tests. The virtual memory system uses a portion of disk, called SWAP, as though it were RAM memory. Virtual memory is organized into “pages,” typically 4096 bytes per page. This size varies greatly from system to system.

A machine that frequently runs out of virtual memory often requires more RAM. Another solution may be to off-load some of its work to other, less burdened systems. When the number of available virtual memory pages breeches an alarm threshold, a TRAP will be sent to the NMS if the NMS has ENABLED this trap (see the TrapManage Group). The TRAP message includes the current number of pages free.

---

Note: The VM group is not available on HP/UX systems.

---

The following table shows the MBUF tests. These tests refer to the BSD UNIX buffer monitoring group. This group is not supported on System V based Operating Systems.

This section details the objects AgentMon can manage (which conform to the MIBII standard.

System Group

This group is used to store basic information about the workstation, who should be contacted, the system's location, and other administrative details.

---

Note: For Solaris only the systems group is implemented. The network statistics normally available on other O/Ss are not available on Solaris.

---

Interfaces Group

These groups list the various network interfaces and information relevant to their current state. They contain various statistics for the different networking protocols that are in use on the server. The information in these groups can help pinpoint host-based network problems, aid in bandwidth utilization, and assist in resource planning. The groups include:

Typically, these groups contain:

The following table shows which MIB II tests AgentMon can manage.

This section details the File subcategory tests. When the file size has breached an alarm threshold, a TRAP PDU will be sent to the NMS if the NMS has ENABLED the TRAP associated with that file. To create a test entry in the testtab file, enter the full pathname to the particular file you want AgentMon to monitor.

File Size

You can use the files group to monitor files. Since many UNIX files can grow without bound, AgentMon provides a method of automatically archiving, truncating, or otherwise averting a file size problem. For example, monitoring the size of sulog and other system files could aid in preserving the security of your system.

For example:

/var/adm/syslog/syslog.log
  :on:testfreq=1:alarmfreq=60:\
  :command=/opt/ENLIGHTEN/example.sh:\
  :!log:high=1000000:units=bytes:\
  :/*end /var/adm/syslog/syslog.log */:

If a test name is a full pathname and it does not refer to a directory, the test is assumed to refer to a file of the same name. The size of the file is then monitored and compared with alarm thresholds. If the named file does not exist, the test is ignored (until the file does exist).

File Accessed

Tests the “last-accessed” time of a file. AgentMon checks the last-accessed time and compares it with the last test's last-accessed timestamp. If there is any change in the last access timestamp by any value in any of the threshold types, AgentMon will issue an alarm.

Unit of measure for test: time

For example:

/etc/shadow accessed | :\
  :on:testfreq=1:alarmfreq=1:log:delta=1:\
  :+jump=1:
  :/*alarms if anyone reads the file */:

File Modified

Tests the “last-modified” time of a file. AgentMon checks the last-modified time and compares it with the last test's last-modify timestamp. If there is any change in the timestamp as compared with any of the tests threshold types, AgentMon will issue an alarm.

Unit of measure for test: time

Since timestamps are in UDT format, any change in the timestamp means that the file has been modified or accessed. Therefore, the most effective way to test if a file has been accessed or modified is to set the +jump threshold type to a value=1.

For example:

!/usr/adm/sulog modified | :\
  :on:testfreq=1:+jump=1:\
  :/*end /usr/adm/sulog modified */:

File Clamped

This test evaluates recent additions in ASCII files to string pattern regular expressions. An alarm is generated if one or more of the regular expressions match one or more of the “new” log entries. A “new” entry is any entry added to the log since this test was last run. Up to 32 regular expressions (string matches) can be configured for a single file-clamp test.

The value parameter passed by AgentMon to the process or script in command will be the number of matches for the test. The Total Value parameter will be set to the extended string that matched the regular expression. The actual string returned in Total Value may vary by platform type and OS version. Refer to the regex(3) man page for the platform on which you wish to configure the file clamp test.

Unit of measure for test: matches

For example:

/usr/adm/sulog clamped| :\
  :on:testfreq=5:alarmfreq=5:\
  :regx1 = (root)*(fail)*:high=1: \
  :/*end /usr/adm/sulog clamped */:

The directories group is used to monitor the number of files in a directory. The directory may be a print queue, an email queue, or any other queue where files in transition are temporarily stored. An individual queue can be monitored by watching the number of files and/or by watching the number of “old” files.

Since queues are bottlenecks, it can become a problem if the number of files becomes large because this could then also lead to a shortage of disk space. An email queue that has many old files could mean that a remote site is no longer reachable.

When the number of files in a queue breaches an alarm threshold, a TRAP PDU will be sent to the NMS if the NMS has ENABLED the trap associated with that queue. The TRAP message will contain the name of the queue and the number of (old) files in it.

To create a test entry in the testtab file, use the full pathname of a directory as the test name. Then the number of files in that directory is tracked. By setting a high alarm threshold, you can be notified when queues are getting too big (backlogged).

For example:

/usr/spool/ps|:\
  :on:testfreq=5:alarmfreq=60:mailer=/bin/mail:\
  :age=5:notify=root:!log:high=20:units=old_files:\
  :/*end /usr/spool/ps */:

Since some queues have accounting files (and other non-queued files) in the queue directory, be sure to consider their number when setting alarm thresholds. If the named directory does not exist, the test is ignored. If the test description for this test contains the age parameter, then only files more than age minutes old will be counted.

The processes group measures the number of currently running processes having the specified name. Given a list of processes and alarm thresholds, AgentMon can tell you when your daemons die, your programs terminate, when you don't have enough instances of a program running, how long a process has been executing, and the amount of memory a process has consumed.

In addition to the usual six types of alarms, two others are available: first instance started and last instance terminated. This means if you are monitoring a process called ABC, an alarm will occur when the first instance of ABC starts up, and when the last one terminates.

For each named process, a TRAP PDU will be sent if an alarm threshold for that process is breached and if the NMS has ENABLED the corresponding trap. The TRAP message will include the name of the process that went into the alarm condition and the number of processes with the same name that are currently running.

---

Note: On BSD flavors of UNIX, you must specify the real process name, that is, the name returned by the ps -c command.

---

Processes Instances

To create a test entry in the testtab file, enter the process name (preceded by an exclamation point (!)) you want AgentMon to monitor.

For example:

!syslogd instances|:\
 :on:testfreq=5:mailer=/bin/mail:notify=root:!log:\
 :units=process(es):+jump=1:-jump=1:\
 :/*end!syslogd*/:

An alarm will be issued if any of the following are true:

To use the API tests, follow these steps:

At each testfreq interval, AgentMon will check the file pointed to by filename to see if its modification time has changed. If the file has changed, AgentMon will reread it. For each line it reads, AgentMon will find the value stored in field (column) data and compare it to any alarm thresholds you have defined for this test (api1, api2, ..., api6).

The following is an example for creating an API Group test.

Scenario: You have several databases and want to know when one or more begin to run out of space. Have cron execute a script that will record the free space and database name to a file called dbsizes. Suppose the file looks like:

1289 Kbytes parts.db
9023 Kbytes customer.db
389 Kbytes phones.db

You could create the new test with the following additional parameters:

as shown in the following example:

api1 |:\
  :on:file=/dbsizes:data=f1:label=f3:low=400:\
  :testfreq=1:alarmfreq=60:\
  :mailer=/bin/mail:notify=root:!log:\
  :/*end api1*/:

Use the right mouse button while the cursor is over a host or pool icon to pop up a menu to further manipulate that icon.

The host pop-up menu offers the following choices:

Select the Host Overview option to bring up the Host Overview window. See “Host Overview” for more details on how to use this window. Select the Remove Host from Current Pool option to delete the host from the current pool and thus remove the host icon from the map (if you have the correct permissions to do so).

The pool pop-up menu offers the following choices:

Select the Zoom into Current Pool option to view the contents of the selected network pool.

Select the Delete Pool option to delete the pool definition from your list of configured pools. EnlightenDSM will prompt you to confirm this action.

Select a host or pool icon by left-clicking on it. You can also move icons by clicking and dragging (if you have Modify permissions).

You can view the contents of a pool by double-clicking on the pool icon. The Status Map will be redrawn to represent the configuration of the selected pool.

You can also bring up the Host Overview window for a host by double-clicking on its icon. See “Host Overview” for more details on how to use this window.