Chapter 5. System Performance Visualization Tools

Several three-dimensional (3D) graphical tools are provided with Performance Co-Pilot (PCP) to assist you in visualizing performance on monitored systems. These tools are implemented with and require Open Inventor, a 3D graphics facility. Each tool is completely described by its own man page, accessible through the man command. For example, the man page for the pmview tool can be viewed by giving the following command:

man pmview 

The following major sections are covered in this chapter:

Other PCP visualization tools, such as arrayvis, clustervis, nodevis, procvis, routervis, txmonvis, weblogvis, webpingvis, webvis, and xbowvis, are not described in this chapter. See the arrayvis(1), clustervis(1), nodevis(1), procvis(1), routervis(1), txmonvis(1), weblogvis(1), webpingvis(1), webvis(1), and xbowvis(1) man pages for information about these pmview based tools.

Overview of Visualization Tools

For the most interesting and complex problems in performance management, the volume of available information is daunting. One approach to dealing with the volume and complexity of the information is to employ automated reasoning. Refer to Chapter 6, “Performance Metrics Inference Engine” for a complete description of the pmie tool that provides this capability. Another approach is to harness the considerable potential for human visual processing to absorb, analyze, and classify large amounts of information.

PCP has been developed with an assumption that being able to draw three-dimension pictures of system performance is a critical monitoring requirement, and one that offers vast potential for increased insight and understanding for the person charged with some aspect of performance monitoring and management.

Building on SGI technologies of high-performance 3D graphics at the workstation, OpenGL and Open Inventor, PCP delivers a range of utilities, services, and toolkits that are designed both to provide basic visualization tools and to foster the local customization of value-added tools to meet the needs of end-user application and operational environments.

Key components to this performance visualization strategy are as follows:

  • Time-series strip charts with pmchart (described in Chapter 4, “Monitoring System Performance”) that allow performance metrics from multiple hosts and multiple Performance Metric Domains (PMDs) to be concurrently displayed on a single correlated time axis.

    Predefined chart configurations for common performance scenarios are provided.

  • Basic three-dimension models for the following:

    • Per-processor CPU utilization with mpvis

    • Per-disk spindle activity with dkvis

    • NFS request traffic with nfsvis

    • MPI function activity with mpivis

  • A generalized, three-dimension performance model viewer, pmview, that can easily be configured to draw scenes animated by the values of arbitrarily selected performance metrics. Tools like dkvis, mpivis, mpvis, and nfsvis are front-ends that create scene descriptions to be displayed and animated with pmview.

  • Icon-sized stripcharts, meters, and indicator LEDs that may be combined into a desktop control and indicator panel using pmgadgets; see “The pmgadgets Command” in Chapter 4.

    The pmgsys command provides a standard layout for important IRIX performance metrics using pmgadgets. The color or height of each gadget is modulated by user-selected performance metrics from one or more PCP sources.

When combined with the VCR and PCP archive services, these visualization tools provide both real-time and retrospective analysis of system performance at many different levels of detail.

The dkvis Disk Visualization Tool

The dkvis tool is a graphical disk device utilization viewer, displaying a bar chart showing disk activity. When you give the dkvis command, you see a bar chart displaying activity on each disk on the monitored system. You see a Total Disk I/O Rate for Host host window similar to the one shown in Figure 5-1.

Figure 5-1. dkvis Total Disk I/O Rate Window

dkvis Total Disk I/O
Rate Window

Each row of blocks on the base plane represents the group of disks connected to a single disk controller (or host adaptor or SCSI bus). The label for each row is generated from the characters common to the names of all of the disks on the controller. For example, in Figure 5-1, the disks in the row labeled dks56 (the same row as the selected block for dks56d2) are dks56d1, dks56d2, dks56d3, and dks56d4.

The dkvis implementation uses the generalized 3D performance viewer pmview as described in “The pmview Tool”. Hence, the command line options for dkvis include the common ones for pmview.

dkvis normally displays the total number of I/O operations per second (IOPS). The -r option may be used to restrict the display to just the read operations or -w may be specified for just the writes.

The dkvis command expresses the utilizations in the information window as percentages of some maximum expected rate (clipped to 100%). The -m flag allows you to override the default maximum value. This is useful if all of the utilizations are small compared to the maximum. In such a situation, specifying a smaller maximum has the effect of magnifying the differences between the blocks. Similarly, if some of the blocks are almost always at full height, there is a good chance that they are being clipped.

A suitable value for the -m option can be determined by clicking the blocks in question, observing the values displayed in the information window for a while, and adding about 10% to the highest value observed. Interactive adjustment of the block height is available via the scale thumb wheel in the pmview viewer.

Complete information on the dkvis command is available in the dkvis(1) man page. The PCP Tutorial contains additional examples on the use of dkvis.

The mpvis Processor Visualization Tool

The mpvis tool is a graphical multiprocessor activity viewer, displaying a bar chart that shows processor activity. When you enter the mpvis command, you see a bar chart displaying activity on each processor on the monitored system. You see a CPU Utilization for Host host window similar to the one shown in Figure 5-2.

Figure 5-2. mpvis CPU Utilization Window

mpvis CPU Utilization 
 Window

This figure shows mpvis monitoring a machine with four CPUs. CPU is spending 80% of its time processing user code and about 20% of its time executing system code. Another CPU is executing user code for close to 90% of the time. The remaining two CPUs are idle.

The display contains five labeled rows of blocks, which represent the breakdown of the activity of a single CPU into five states. There is one column of five blocks for each CPU on the system being monitored. These five states are as follows:

idle 

No activity

wait 

Like idle but waiting for I/O

intr 

Processing an interrupt

sys 

Executing in the IRIX kernel

user 

Executing user code

The mpvis implementation uses the generalized 3D performance viewer pmview as described in “The pmview Tool”. Hence, the command line options for mpvis include the common ones for pmview.

Complete information on the mpvis command is available in the mpvis(1) man page. The PCP Tutorial contains additional examples on the use of mpvis.

The nfsvis NFS Activity Visualization Tool

The nfsvis tool is a graphical NFS (Network File System) activity viewer, displaying a bar chart that shows NFS request activity on the monitored system. NFS is optional software, and may not be present on all systems or at all sites.

When you run the nfsvis command, you see a bar chart displaying NFS load on the monitored system. You see a NFS Client V2 & Server V2 Request Traffic for host host window similar to the one shown in Figure 5-3.

Figure 5-3. nfsvis NFS Client V2 & Server V2 Request Traffic Window

nfsvis NFS Client
V2 & Server V2 Request Traffic Window

The statistics are broken into two groups:

Client 

Requests made by the monitored machine to NFS servers on other machines

Server 

Requests from other machines for the NFS server on the machine being monitored

The statistics in each of these two groups are the same, except that the client group is for outgoing requests and the server group is for incoming requests. Within each group, the requests are further broken down into three categories:

  • Requests relating to data within files

  • Requests for directory operations (for example, to rename a file)

  • Requests involving other attributes of files

The nfsvis implementation uses the generalized 3D performance viewer pmview as described in “The pmview Tool”. Hence, the command line options for nfsvis include the common ones for pmview.

Complete information on the nfsvis command is available in the nfsvis(1) man page. The PCP Tutorial contains additional examples on the use of nfsvis.

The mpivis MPI Function Activity Visualization Tool

The mpivis tool displays a 3D bar chart for the rate of elapsed time used by a particular set of MPI functions for an MPI application. Figure 5-4 shows a sample mpivis MPI Activity for ASH ash on Host host display.

Figure 5-4. mpivis MPI Activity Window

mpivis MPI Activity Window

This mpivis display was produced with the following control file:

# label     MPI function  color
#
send        MPI_Send      blue
recv        MPI_Recv      orange
bcast       MPI_Bcast     burlywood
barrier     MPI_Barrier   violet
allreduce   MPI_Allreduce turquoise

From Figure 5-4, you can see that the MPI application has 4 ranks and that the MPI_Allreduce and MPI_Barrier functions have recently consumed some time, as opposed to MPI_Bcast, MPI_Recv, and MPI_Send. The other category, which is always yellow, shows that some other MPI functions are also active. Ranks 0 and 1 have consumed about half as much time in MPI_Allreduce and MPI_Barrier than ranks 2 and 3. The chart prisms of ranks 2 and 3 of MPI_Barrier have turned white. This indicates that they have surpassed the maximum default value of 300 milliseconds per second.

The state row of cylinders at the back does not grow; its color changes to indicate the currently executing MPI function at the last update of the metrics. In Figure 5-4, the state is green, which is a reserved color for applications.

The weblogvis Visualization Tool

The weblogvis tool displays a 3D bar chart of data extracted from Web-server log files.

For each server, weblogvis displays idle time, total activity, and the activity classified by the size of the resultant request. The activity is, by default, the request rate (requests per second) as shown in Figure 5-5, but optionally a data rate (bytes per second) can be requested as shown in Figure 5-6. If the server logs report caching statistics, the bar displays are shown as a stack of values representing cached and non-cached results. Two examples of suitable caching log files are the Netscape common extended log format and the Squid default log file format.

Figure 5-5. weblogvis Display of Request Rate Classified by Request Size

weblogvis Display of Request
Rate Classified by Request Size

Figure 5-5 shows a host with four servers, one without cache statistics, two using the common extended log format, and one Squid server. The scene is comprised of three objects:

Idle time per server 

This is a single-value bar representing the number of seconds that the log file has been unaltered.

Total activity rates per server 

For servers that report extended statistics (local1 , proxy1, and squid1), the total is composed of three stacked values representing the activity rates satisfied by the browser cache (yellow), by the proxy cache (blue) or by a remote server (red). For server logs without the extended data ( cern1), the single value (red) represents the total activity rate.

Activity rates per server classified by request size 

For servers that report extended statistics, each bar is composed of two stacked values, representing the activity rates for that request size, that are satisfied by the proxy cache (blue) or by a remote server (color coded based on the request size). For server logs without the extended data, the single color-coded value represents the activity rate for that request size.

Figure 5-6 shows the same host with an alternate type of display. It shows the activity rates classified by the request type instead of the request size.

Figure 5-6. weblogvis Display Request Rate Classified by Request Type

weblogvis Display Request Rate
Classified by Request Type

The pmview Tool

The pmview tool is a generalized 3D Open Inventor application that supports dynamic displays of clusters of related performance metrics as utilization blocks (or towers) on a common base plane. The pmview tool is the basis for dkvis, mpvis, osvis, nfsvis, mpivis, and weblogvis all discussed in this chapter, as well as arrayvis , clustervis, nodevis, procvis, routervis, txmonvis, webpingvis, webvis, and xbowvis . The pmview tool may also be used to construct customized 3D performance displays.

A closely related tool to pmview is pmview+. It has the same capabilities as pmview, as well as the support for showing interconnects between the bases of pmview grids. It was written to support oview in its display of SGI 3000 servers.

Open Inventor is an object-oriented toolkit that simplifies and abstracts the task of writing graphics applications into a set of easy-to-use objects. Its run-time support is distributed with IRIX system software in the inventor_eoe.sw product image.

The pmview command displays performance metrics as colored blocks arranged in a grid on a grey base plane. The height of each block changes as the value of its corresponding metric (or metric instance) changes. Labels may be added to the scene to help identify groups of metrics, as shown in Figure 5-1, Figure 5-2, and Figure 5-3.

A configuration file is used to specify the position, color, and scale of metrics and metric instances in the scene. Metric values that exceed the associated scaling factor are displayed at the maximum height and change color to white. If a metric is unavailable, the bar height is minimized, and the bar color changes to grey.

Normally, pmview operates in live mode where performance metrics are fetched in real time. The user can view metrics from other accessible Internet hosts that are running the PCP collector daemon, pmcd. The pmview tool can also replay archives of performance metrics collected by pmlogger.

All metrics in the PMNS with numeric value semantics from multiple hosts or archives may be visualized. The pmview tool examines the semantics of the metrics and, where sensible, converts the fetched metric values to a rate before scaling.

The pmview tool window contains a menu bar, time and scale controls, metric and time values, and an examiner window; see the ivview command, which displays the 3D scene.

The left, right, and bottom edges of the examiner viewer window contain a variety of thumb wheels and buttons that allow the user to adjust the visualization of the 3D scene. The Rotx and Roty thumb wheels allow the user to rotate the scene about the X and Y axes, respectively. The Dolly thumb wheel moves the virtual camera closer to or further from the scene, allowing the user to examine specific parts in detail or view the entire scene.

On the right edge of the viewer are eight buttons that affect the way the user can interact with the scene:

  • The pointer button changes the cursor to a pointer that allows blocks in the scene to be selected. The Esc key can also be used to toggle between the pointer and hand cursors.

  • The hand button changes the cursor to a hand that can be used to rotate, translate, and examine the scene via Dolly, using a combination of mouse buttons and movement.

    The left mouse button can be used to rotate the scene in the direction of the mouse. Releasing the mouse button before the mouse has stopped moving causes the scene to continue rotating until a mouse button is pressed again.

    The middle mouse button can be used to pan the scene. By pressing both left and middle buttons, the mouse can be used as a virtual camera.

  • The question mark button displays SGI Help for the examiner viewer. To install online help, use inst to install the inventor_eoe.sw.help package from your IRIX system software distribution. See the Performance Co-Pilot release notes for more information on prerequisite subsystems.

  • The home button changes the scene back to its original position, or the position set by the home pointer button.

  • The home pointer button sets the new home position of the scene to be the scene currently in view.

  • The eye button resizes the scene so that it completely fits into the 3D viewing area.

  • The cross-hairs button moves the scene so that the object under the cursor is in the center of the viewing area. Change the hand cursor and press the cross-hairs button. The cursor changes to a target. Select the block to be centered and the scene rotates and translates appropriately.

  • The perspective box button switches between perspective and orthogonal projections.

Pressing the right mouse button within the scene displays a menu of options that affect how the 3D scene is drawn. The options include drawing the blocks as wireframes and turning on stereo viewing.

When the pointer cursor is active, more information about the 3D scene can be obtained. Text describing the metric represented by the block beneath the cursor displays in the top text box of the pmview window. This text displays the source, name, and instance of the performance metric, and the value, units, and percentage of the expected minimum the value represents.

Clicking the left mouse button on a block highlights the block with a red wireframe, as shown in Figure 5-7. The metric description text box is now fixed on that metric and the values continue to be updated as new metrics are fetched. This allows other actions to be performed with the mouse while examining a single metric in detail at the same time. Click the left mouse button on the space surrounding the scene to remove the selection.

Figure 5-7. pmview Window with a Block Selected

pmview Window with a Block Selected

Multiple blocks may also be selected by either Shift clicking with the left mouse button or by clicking on a base plane. Shift clicking toggles the selection status of a particular block and leaves the selection status of other blocks unaltered. Clicking on the base plane selects all blocks belonging to the base plane. Whenever multiple blocks are selected, no accompanying text is displayed in the text box. However, multiple block selection affects the launching of other tools because all metrics on the base plane are considered to be selected as a group.

pmview Menus

There are four menus in pmview tools:

File 

Records, saves, and prints scenes

Options 

Accesses the time controls

Launch 

Starts other tools

Help 

Obtains online help

The Launch menu consists of a list of tools that operate on the current selection of metrics. How the tool is invoked depends on the type of tool. Tools that operate on any metric (such as pmchart , pmval, and pmdumptext) use the metrics selected directly as input. Thus pmchart displays all the selected metrics in a chart, pmval is invoked within winterm for each metric, and pmdumptext displays multiple metrics in one winterm. Other tools use what metrics are pertinent. If no metric is pertinent or selected, only the source of the metrics is used, that is, the monitored host or archive. For each Launch menu item there is an associated launch script. The launch scripts generally know the relationship between routers, nodes, and CPUs. Thus if CPUs are selected in mpvis and if nodevis is launched, only the nodes that have the selected CPUs attached are displayed.

Some launchable tools are listed below:

dkvis, mpvis, nfsvis , and osvis  

pmview-based tools for visualizing disk activity, CPUs, NFS, and the OS (operating system).

pcp 

Brings up a window that summarizes the PCP installation.

pmdumptext 

Brings up a window that shows the performance metrics as text.

pmchart 

A tool for graphically displaying and correlating time-series trends of performance metrics. See “The pmchart Tool” in Chapter 4, for details.

pmgsys 

A miniature IRIX performance metrics viewer, available only in live mode, not in archive mode.

pmkstat 

A text-based tool that displays, at intervals, a high-level summary of system performance.

pmval 

A tool that displays the values of performance metrics textually. Only one metric (with one or more instances) may be selected to successfully launch this tool.

In addition to the menu options for time controls, the current direction and mode of the time controls is shown in a button in the top-left corner of the pmview window. Pressing this button displays the time control dialog.

Below this button is a thumb wheel and an editable text box to specify a scale multiplier that is applied to all values in the scene. Spinning the thumb wheel to the right, or incrementing the value in the text field, increases the scaling and raises the height of the bars. Conversely, spinning the thumb wheel to the left or decrementing the text field decreases the scaling and lowers the height of the bars.

The button beside the thumb wheel resets the scale to one. This is especially useful when the scale specified in the configuration file reduces the usefulness of the visualization as a consequence of the bars being either too low or beyond the maximum scale height.

Creating Custom Visualization Tools with pmview

At startup time, a configuration file is read that specifies the following:

  • Geometry for the scene to be displayed by pmview

  • Associations between the visual appearance of “blocks” and performance metrics

The scene is based on a grid that can contain a variety of objects and can resize itself to accommodate objects of varying sizes. To distinguish this configuration file format from an earlier (still supported) format, configuration files must begin with the following line:

pmview Version 2.1 

All lines beginning with a # character are treated as comments and ignored. Spaces, tabs, and newlines are treated as white space to allow multiple statements on the same line. The simplest configuration file consists of a single object that may represent one or more metrics and metric instances.

The configuration file consists of two sections: global parameters and color lists, and the object definitions. The global parameters control the size of the objects in the scene. For example, a scaling factor of 1.2 can be applied to all objects with the following line:

_scale 1.2

Groups of colors may be associated with a name and referenced later in the file. Colors may be X(1) color names, X(1) numerical colors, or three real values representing the saturation of red, green, and blue, respectively. The following color list contains three identical colors:

_colorlist cpu ( red rgbi:1.0/0.0/0.0 1.0 0.0 0.0 )

The mpvis configuration file (which can be generated with the -V option) looks like Example 5-1:

Example 5-1. mpvis Configuration File

pmview Version 2.1 "mpvis" "-V" "-C"
#
# mpvis
#
# ncpus = 1
# nrows = 1
# ncols = 1
#
# List:
# cpu0
_gridSpace 120

_colorlist cpu ( green2 cyan2 yellow2 red2 blue2 )
_grid 0 0 _hide ( # outer grid
        _baseLabel "SGI PCP : CPU Utilization for Host ha1.melbourne.sgi.com\ncpu0 only"
        _bar _groupByInst (
            _metrics (
                kernel.percpu.cpu.idle[cpu0]        1000 "idle"
                kernel.percpu.cpu.wait.total[cpu0]  1000 "wait"
                kernel.percpu.cpu.intr[cpu0]        1000 "intr"
                kernel.percpu.cpu.sys[cpu0]         1000 "sys"
                kernel.percpu.cpu.user[cpu0]        1000 "user"
            )
            _colorlist cpu
            _baseLabel "SGI PCP : CPU Utilization for Host ha1.melbourne.sgi.com\ncpu0 only"
        )
)

Multiple objects can be visualized using a _grid object, which may contain multiple objects (including more _grid objects). The _grid object resizes columns and rows to accommodate the largest contained object. Objects can occupy multiple grid squares and can be aligned with a particular edge or corner of a grid square. The _bar object has a single bar for each metric instance and labels for each metric. The scale height for each metric instance is 1000 in the units of the metric (milliseconds utilization per second).

The specification file shown in Example 5-2 produces a scene like the one shown in Figure 5-8. The file has a grid, labels, bars, and a stack utilization object.

Example 5-2. Specification File for pmview

pmview Version 2.1
_colorlist cpu_colors ( blue2 red2 yellow2 cyan2 green2 )
_colorlist disk_colors ( purple2 yellow2 )
_colorlist memory_colors ( rgbi:1.0/1.0/0.0 rgbi:0.0/1.0/1.0 rgbi:1.0/0.0/0.0
                              rgbi:1.0/0.0/1.0 rgbi:0.0/0.0/1.0 rgbi:0.0/1.0/0.0 )
_grid hide (
_label 3 1 _west _down _large “CPU”
       _stack 4 1 _west _utilmod (
             _metrics (
               kernel.all.cpu.user        1000
               kernel.all.cpu.sys         1000
               kernel.all.cpu.intr        1000
               kernel.all.cpu.wait.total  1000
               kernel.all.cpu.idle        1000
           )
           _colorlist cpu_colors
           _baseLabel “CPU Utilization”
       )
_label 3 3 _west _down _large “Load”
       _bar 4 3 2 1 _west (
           _metrics (
               kernel.all.load[15]    2
               kernel.all.load[5]     2
               kernel.all.load[1]     2
           )
           _metriclabels _away ( “15” “5” “1” )
           _colorlist ( blue2 blue2 blue2 )
_baseLabel “Average System Load over last 1, 5 and 15 minutes\nNormalized to 2”
       )
_label 0 1 _west _down _large “Mem”
         _stack 1 1 _west _utilmod (
           _metrics (
               mem.util.kernel        1
               mem.util.fs_ctl        1
               mem.util.fs_dirty      1
               mem.util.fs_clean      1
               mem.util.user          1
           )
           _colorlist memory_colors
           _baseLabel “Physical Memory Utilization”
       )
       _label 0 3 _down _large “Disk”
       _stack 1 3 _west _cylinder (
           _metrics (
               disk.all.read          100
               disk.all.write         100
           )
           _colorlist disk_colors
           _baseLabel “Disk Operations\nNormalized to 100 I/Os per second”
        )
)


Figure 5-8. Custom pmview Scene

Custom pmview Scene

To assist in the creation of front-end tools, a file containing shell procedures for generating usage information and parsing pmview command line options is located at /usr/pcp/lib/pmview-args . The dkvis, mpvis, nfsvis, and osvis tools are all shell scripts that use these shell procedures to generate a configuration file for pmview.