Chapter 2. Collecting Event Data

An event is any action by a thread, process, or hardware component that could affect the state of the system. IRIXview lets you collect event data on a selected IRIX target system or open previously saved files containing event data. These saved files are called event logs. This chapter covers the following topics:

Using IRIXview to Collect Event Data

With IRIXview, you may collect event data using the Target window. This section tells you how, and contains the following topics:

Prerequisites for Collection

To collect event data, the the target system must be running the rtmond daemon; see rtmond(1). This daemon is default-installed as part of the eoe.sw.perf subsystem, but might need to be turned on with chkconfig; see chkconfig(1M).

To collect event data from a remote host, you must edit the /etc/config/rtmond.options file on that remote host and delete the -a localhost access security flag.

Using the Target Window

To open the Target window, choose Windows > Target from the IRIXview main menu. Figure 2-1 shows the Target window.

Figure 2-1. Target Window


Table 2-1 explains the event classes that can be checkmarked in the Target window.

Table 2-1. Event Classes in Target Window

Event Class

Explanation

Debug

Kernel debug events (not available on production systems).

Syscall

System calls, documented in section 2 of the reference pages.

Signal

Signal delivery and reception; see the signal(5) reference page.

Tasks

User process and thread scheduling.

Intr

Hardware interrupts; see the intr(D2) reference page.

Framesched

Frame Scheduler (FRS) operations; see the FRS(3) reference page.

Profile

Kernel profiling events (not available on production systems).

VM

Virtual memory operation.

Disk

I/O activity to and from disk drives.

Netsched

Network I/O scheduling

Netflow

Network I/O flow

Alloc

Memory allocation events; see the brk(2) reference page.

Follow these steps to collect event data using the Target window:

  1. Type the target system's name in the Host Name field, or accept the localhost default The default hostname is localhost, which automatically selects the system on which you started irixview.


    Note: To determine a system's host name, on the target system enter either hostname or uname -n.


  2. Enter the CPU for which you want to collect data, or use the default of 0- to indicate all CPUs on the system specified in the Host Name field. You can separate multiple CPU numbers with commas; for example: 0, 3. You can also enter a consecutive series of processors separated with a dash; for example: 1-3.

  3. Select which events you want to collect, or accept the default selections. The All button selects all of the event check boxes, and None removes all selections. Click Signal, VM, or Disk to automatically select all events related to such categories. The check boxes add events to the current selection.


    Note: You must be superuser to collect Syscall events; otherwise, a warning message appears, the program deselects Syscall, and event logging continues.


  4. Optionally, enter the maximum amount of time (in seconds) or number of events for which you want to collect data, then click Start. Data collection continues for the amount of time or number of events you specified. If you do not specify an amount of time or number of events, data collection continues until you click Stop.

  5. It is a good idea to save the data collected during this event logging session. To do this, choose File > Save. You lose event data from the current session if you fail to save before you open an event file, collect new data, or exit IRIXview.

  6. Now you can display the event data; see Chapter 3, “Displaying Event Data.”

Troubleshooting Data Collection

Some problems you might encounter with data collection include:

  • Host connection could not be established.

  • Context View graph does not update.

  • “Lost Event” messages appear in the IRIXview main window.

To collect event data, the rtmond daemon must be running on the target system; see rtmond(1). Normally, this daemon starts automatically, but if you are having problems collecting data, make sure the daemon has actually started by running the ps command on the target system; see ps(1).

When you are trying to collect data from a remote host, you must have been given permission to obtain event data. Ensure that the /etc/config/rtmond.options file on the remote target system has been stripped of its -a localhost flag.

If you are accustomed to seeing the Context View graph update during data collection, please note that this does not occur in IRIXview, as it did in previous releases.

If you click the Start and Stop buttons in the Target window, but event data never appears in the graph window and the main window lists “Lost Event” error messages, the problem could be that the event rate exceeds the bandwidth of the connection, so the event logging mechanism shuts itself off. Try one or more of the following strategies to solve the problem:

  • Close all graphs while you are collecting data.

  • If you believe that network traffic may be the cause of the problem, isolate the host and target on a subnetwork or a standalone network.

  • Use the rtmon-client tool to collect the event data; see “Using rtmon-client or par to Collect Event Data”.

Using rtmon-client or par to Collect Event Data

The rtmon-client tool offers a command-line alternative to IRIXview for event collection. The rtmon-client tool allows you to collect an event log on the host without loading it into IRIXview until some later time. For example:

  • Your system generates a large amount of event data, and rtmon-client imposes less processor overhead than IRIXview. Because IRIXview must process event data before it can display the data, you might encounter target event buffer overflow conditions if your network is not fast enough to allow IRIXview to process events in real time (see “Troubleshooting Data Collection”).

  • You want to collect event data at a remote site for later analysis at a lab. You can use rtmon-client to collect an event log as the remote system runs, and then import and view the event data with the IRIXview GUI at the lab.

  • You want to save event logs to multiple files, each generated from a different operating condition. The rtmon-client tool provides an option that saves event logs into multiple files; each time event logging is turned on, a new file is generated.

System prerequisites are the same for rtmon-client and par as they are for the Target window. See “Prerequisites for Collection” for details.

Using rtmon-client for Collection

To use rtmon-client, enter the following on the target system:

rtmon-client 

By default, the collected event log is saved in the file default. You can change this name with the -f option. To stop rtmon-client event data collection, press Ctrl+C. Additional options specify the number of seconds to run (-t sec), a remote host to monitor (-h name), a processor list (-p cpuList), and whether to enable debugging (-d 1). For a description of rtmon-client and its options, see the rtmon-client(1) reference page.

As an example, to create 10-second traces for processor 1 through 3, and save them to the file mp_test.irv, enter the following:

rtmon_client -f mp_test -p 1-3 -t 10 

This command creates three files, mp_test.1.wvr, mp_test.2.wvr, and mp_test.3.wvr for the three processors.

Using par for Collection

The par command is useful for collecting event data limited to a number of individual processes, rather than for the system as a whole. Here is an example par command:

par -s -SS -O /usr/tmp/test.ivr -o /dev/null -p ProcessID -p 2ndProcessID 

In this example, -s means collect system call and signal data, -SS means print both the summary of system call activity and a trace of each system call and signal action, -O outputs events to file /usr/tmp/test.ivr, -o disposes of other output to /dev/null, and the two -p options indicate process IDs to trace.

The par command exits when all its given processes terminate, or you can press Ctrl+C to terminate par manually. For more information see the par(1) reference page.

Adding Timestamps to Your Program

Use the user timestamp logging function rtmon_log_user_tstamp() to insert timestamps into application code. This function logs events and passes them to the rtmond daemon. Such user events can be viewed in a Context View graph, shown by the symbol in Figure 2-2, and dragged and dropped into the Event Inspector window. For more information about implementing the timestamp logging function, see the reference page for rtmon_log_user_tstamp(3).

Figure 2-2. User Event Symbol


User events are merged into the standard system event data stream (it is really the same stream), so user events appear in chronological order along with system events.