Chapter 4. Performance Management with Performance Co-Pilot (PCP)

Performance Co-Pilot (PCP) is a framework and set of services that support system-level performance monitoring and performance management.

In the ISE product, the PCP configuration has been customized to produce Web-based reports generated by PCP tools running in the background. These reports are available from the SGI Internet Server Web administration GUI. These reports are updated in real-time to display short-term and long-term Web server activity and utilization.

This chapter discusses the following:

Performance Status Reports

This section discusses the following:

Accessing the Latest Reports

To access the performance status reports, see the Monitoring section of the SGI Internet Server Web administration interface using the following URL:

http://hostname/sgi-iserver/pcp/management.html

From the Monitoring section, you can select one of the following:

  • last hour displays detailed performance statistics over the last hour. This report is updated every minute.

  • last day displays a summary of statistics over the last 24 hours. This report is updated every 15 minutes.

You can access the latest reports directly by using the following URLs:

http://hostname/sgi-iserver/pcp
http://hostname/sgi-iserver/pcp/ISE-1hour.gif
http://hostname/sgi-iserver/pcp/ISE-24hour.gif

Accessing Older Reports

The status reports are placed in the following directory:

/home/httpd/html/sgi-iserver/pcp

This directory contains all existing previous reports, which include the hour or day of the week in the file name. The hourly images are named as follows:

  • snap_hHH.gif for the hourly images, where HH is a value from 00 (midnight) through 23 (2300 hours, or 11:00 PM). For example, snap_h18.ISE-24hour.gif was created at 6:00 PM. The file will contain data from the previous hour.

  • snap_dDD.gif for the daily images, where DD is a value from 00 (Sunday) through 06 (Saturday). For example, snap_d03.ISE-24hour.gif was created on Wednesday. The file will contain data from the previous 24 hours.

There will be 24 ISE-1hour.gif files, the oldest of which is overwritten every hour. There will be 7 ISE-24hour.gif files, the oldest of which is overwritten every 24 hours.

Report Contents

Figure 4-1 shows an example report.

Figure 4-1. Example Last Day Report

These images show various charts with a common time-axis. The following subsections describe each chart.

Per Server Data Transfer Rate

The Per server data transfer rate chart shows the data throughput for each www and ftp server configured on the system as a stacked bar chart. The total height of the bars represents aggregate data throughput for all servers on this host. The chart provides automatic scaling, hence the units on the Y-axis may represent a scale from bytes to Gbytes according to the current traffic.

IO Operations Per Device

The IO Operations Per Device chart shows disk input/output (I/O) operations per disk device. All SCSI and IDE disks are included in this chart. The rate of disk I/O operations is strongly correlated with device throughput and the I/O rate may be used as a general indication of device saturation. A high I/O rate on a particular device while others remain relatively idle indicates an imbalance in the I/O load across all devices.

Disk Throughput on Controller controller_name

The Disk Throughput on Controller controller_name chart shows actual data transfer rates for each SCSI device, with one chart for each SCSI controller in the system. Each chart shows a stacked bar, with one component for reads and one component for writes, repeated for each device on the particular SCSI controller. The total height of the stacked bar indicates the total disk traffic throughput on that controller.

A particular controller showing a lot of traffic while others remain relatively idle indicates an imbalance in the I/O load across the system. If disk throughput is perceived to be a performance bottle-neck, an improvement may be gained by spreading the load more evenly across the available controllers; that is, by moving one or more disks to a different controller or moving one or more filesystems to different disks.

Network Performance

The Network Performance chart shows the rate of errors, collisions, and dropped packets for each network interface configured in the system (other than serial/modem lines). The Transmission Control Protocol (TCP) packet retransmission rate is also shown in total for all network interfaces.

A high rate of collisions indicates contention for local area network bandwidth with other systems. It is normal to see a small number of collisions, usually in bursts. However, you should investigate a sustained, high rate of collisions.

High rates of dropped packets, errors, or TCP retransmissions indicate poor Web server performance and/or network problems requiring investigation.

Network Throughput on interface_name

The Network Throughput on interface_name chart shows actual network transfer rates for each network device (other than serial/modem interfaces), with one chart for each network interface configured in the system. Each chart shows both incoming and outgoing traffic as a stacked bar, with the total height of the bar indicating total throughput. The following indicate problems:

  • Sustained traffic near the available bandwidth indicates that the system is overloaded by client connections. In this situation, most users will be experiencing poor service.

  • Unexpectedly low traffic indicates that there may be failing network cables and/or hardware errors. In this situation, users will be experiencing very poor service or no service at all.

PCP Administration

This section discusses the following:


Note: The PCP configuration for ISE requires the following packages, which are preinstalled on your system:

  • pcp 2.1.3-1 or later

  • pcp-pro 2.1.2-2 or later

  • pcp-ise 1.0.0-2 or later



The PCP application generating the report images also requires that the XFree Virtual Framebuffer Server (XFree-xvfb*.rpm) is installed; XFree is provided with most base Linux distributions.


Changing the Reports

You can add new reports or change the report contents by adding or changing files ending in .options in the /var/pcp/config/pcp-ise directory. Each file ending with the .options suffix specifies the command line options that are passed to pmchart(1), which is the tool used to generate the images. In particular, you can use the -h option to tell pmchart to monitor a remote host (such as another Web server in your environment that has the PCP RPMs installed). The available options for pmchart are fully described in the pmchart(1) man page.


Note: If you use the -h option to pmchart(1), you must open up the appropriate ports in your ipchains(8) configurations on both systems. In addition, note that the PCP reports produced with the Web administration GUI will always feature the name of the local systems. Because this could cause confusion, SGI does not recommend using the -h option with the GUI.

If you add, change, or remove files from the /var/pcp/config/pcp-ise directory, you must restart pcp-ise with the following command:

# /etc/rc.d/init.d/pcp-ise restart 

Changing the Web Configuration

If you change your Web configuration in any of the following ways, you must reinstall the weblog Performance Metrics Domain Agent (PMDA) and restart the pcp-ise package:

  • Add a Web or ftp server

  • Delete a Web or ftp server

  • Relocate server activity logs

To reinstall weblog and restart pcp-ise, enter the following (in the bash(1) shell):

# cd /var/pcp/pmdas/weblog
# export QUIET_INSTALL=Y
# ./Install
# /etc/rc.d/init.d/pcp-ise restart


Real-Time and Remote Display

The performance reports produced by PCP for ISE are configured to work "out of the box", with no user setup normally required. However, you can also use PCP interactively to examine the performance of your Web serving environment in real-time.

You can use PCP interactively by remotely logging in to the SGI Internet Server and displaying back the monitoring information to your remote workstation, as shown in Figure 4-2.

Figure 4-2. Real-Time and Remote Display

For more information about using PCP interactively, see the documentation listed in “For More Information about PCP”.

Real-Time and Remote PCP Installation

You may want to install the pcp, pcp-ise, and pcp-pro packages on a workstation in addition to the SGI Internet Server. Doing so will result in better display performance, especially if the workstation has hardware accelerated graphics. Figure 4-3 shows this deployment.

Figure 4-3. Real-Time and Remote PCP Installation



Note: PCP also requires that the XFree Virtual Framebuffer Server (XFree-xvfb*.rpm) is installed on the workstation; XFree is provided with most base Linux distributions.

To install the required packages on a Linux workstation, do the following:

  1. Log into the workstation as root.

  2. Insert the ISE CD-ROM in the workstation's driver.

  3. Mount the CD-ROM by entering the following:

    # /bin/mount /dev/cdrom /mnt/cdrom

  4. Change to the /mnt/cdrom directory:

    # cd /mnt/cdrom

  5. Enter the following to install the packages :

    # rpm -i /mnt/cdrom/RPMS/XFree86-Xvfb*.i386.rpm
    # rpm -i /mnt/cdrom/RPMS/pcp*.i386.rpm

After the RPMs are installed, the PCP daemons will start automatically on each reboot.


Note: To manually start the PCP daemons, enter the following:
# /etc/rc.d/init.d/pcp start
# /etc/rc.d/init.d/pcp-ise start



To monitor other performance metrics, you can use live-mode PCP tools such as pmchart(1) and pmgsys(1). For the Web-centric setup, weblogvis(1) and webvis(1) provide 3D visualizations of server resource utilization and Web server activity.

You can use the configuration file for pmchart to display the same information as shown on the Web page; you can also use it to monitor the metrics at a different sampling rate or do live monitoring. Use the following command:

$ pmchart -c ISE

The PCP monitoring tools that use a 3D display (mpvis, dkvis, osvis, webvis, and weblogvis) will work best if there is a 16 bpp or better visual available. An 8 bpp visual will still work, but some annoying colormap flashing may occur. You can use the xdpyinfo tool to determine what visuals are available.

For More Information about PCP

Refer to http://oss.sgi.com/projects/pcp for additional information. Also see the following README files:

/usr/doc/pcp-Version/README
/usr/doc/pcp-ise-Version/README
/usr/doc/pcp-pro-Version/README

If you want to use PCP for interactive performance analysis, see the Performance Co-Pilot User's and Administrator's Guide. If you want to develop your own performance monitoring tools or agents, read the Performance Co-Pilot Programmer's Guide. Both of these books are available online from the SGI Technical Publications Library: http://techpubs.sgi.com.

These documents describe the IRIX version of the PCP product, which differs only slightly from the Linux version. Table 4-1 summarizes the differences.

Table 4-1. PCP Differences between IRIX and Linux

Directory

IRIX

Linux

rc/startup scripts

/etc/init.d

/etc/rc.d/init.d

Private PCP binaries

/usr/pcp/bin

/usr/share/pcp/bin

Shared PCP files (shareable for diskless)

/usr/pcp

/usr/share/pcp

Directory of man pages

/usr/share/catman

/usr/man

PCP logs

/var/adm/pcplog

/var/log/pcp

PCP documentation

/var/pcp

/usr/doc/pcp-Version

Directory for PCP demos and examples

/var/pcp/demos

/usr/share/pcp/demos

magic, as used by file(1)

/etc/magic

/usr/share/magic

The version of PCP packaged with the SGI Internet Server includes features that are derived from the IRIX version of PCP. For more information, see: http://www.sgi.com/software/co-pilot.