Chapter 1. Programming Performance Co-Pilot

Performance Co-Pilot (PCP) provides a systems-level suite of tools that cooperate to deliver distributed, integrated performance management services. PCP is designed for the in-depth analysis and sophisticated control that are needed to understand and manage the hardest performance problems in our most complex systems.

PCP provides unparalleled power to quickly isolate and understand performance behavior, resource utilization, activity levels and performance bottlenecks.

Performance data may be collected and exported from multiple sources, most notably the hardware platform, the operating system kernel, layered services, and end-user applications.

There are several ways to extend PCP by programming certain of its components:

In addition, the topic of customizing a PCP installation is covered in the chapter on customizing and extending PCP service in the Performance Co-Pilot for IRIX Advanced User's and Administrator's Guide or in the Performance Co-Pilot for IA-64 Linux User's and Administrator's Guide.

1.1. PCP Architecture

This section gives a brief overview of PCP architecture. For an explanation of terms and acronyms, refer to Appendix A, “Acronyms”.

PCP consists of several monitoring and collecting tools. Monitoring tools such as pmval and pminfo visualize metrics, but have minimal interaction with target systems. Collection tools, called PMDAs, extract performance values from target systems, but do not provide graphical user interfaces.

Systems supporting PCP services are broadly classified into two categories:


Hosts that have the PMCD and one or more PMDAs running to collect and export performance metrics


Hosts that import performance metrics from one or more collector hosts to be consumed by tools to monitor, manage, or record the performance of the collector hosts

Each PCP enabled host can operate as a collector, or a monitor, or both.

There are separate node-locked licenses for additional tools when available on IRIX systems for collector and monitor functions.

Figure 1-1 shows the architecture of PCP. The monitoring tools consume and process performance data using a public interface, the Performance Metrics Application Programming Interface (PMAPI).

Below the PMAPI level is the PMCD process, which acts in a coordinating role, accepting requests from clients, routing requests to one or more PMDAs, aggregating responses from the PMDAs, and responding to the requesting client.

Each performance metric domain (such as IRIX, Linux, or some Database Management System (DBMS)) has a well-defined name space for referring to the specific performance metrics it knows how to collect.

Figure 1-1. PCP Global Process Architecture

PCP Global Process Architecture

1.1.1. Distributed Collection

The performance metrics collection architecture is distributed, in the sense that any monitoring tool may be executing remotely. However, a PMDA is expected to be running on the operating system for which it is collecting performance measurements; there are some notable PMDAs such as Cisco and Array that are exceptions, and collect performance data from remote systems.

As shown in Figure 1-2, monitoring tools communicate only with PMCD. The PMDAs are controlled by PMCD and respond to requests from the monitoring tools that are forwarded by PMCD to the relevant PMDAs on the colllector host.

Figure 1-2. Process Structure for Distributed Operation

Process Structure for Distributed Operation

The host running the monitoring tools does not require any collection tools, including PMCD, since all requests for metrics are sent to the PMCD process on the collector host.

The connections between monitoring tools and PMCD processes are managed in libpcp, below the PMAPI level; see the pmapi(3) man page. Connections between PMDAs and PMCD are managed by the PMDA functions; see the pmda(3) and pmcd(1) man pages. There can be multiple monitor clients and multiple PMDAs on the one host, but there may be only one PMCD process.

1.1.2. Name Space

Each PMDA provides a domain of metrics, whether they be for the operating system, a database manager, a layered service, or an application module. These metrics are referred to by name inside the user interface, and with a numeric Performance Metric Identifier (PMID) within the underlying PMAPI.

The PMID consists of three fields: the domain, the cluster, and the item number of the metric. The domain is a unique number assigned to each PMDA. For example, two metrics with the same domain number must be from the same PMDA. The cluster and item numbers allow metrics to be easily organized into groups within the PMDA, and provide a hierarchical taxonomy to guarantee uniqueness within each PMDA.

The Performance Metrics Name Space (PMNS) describes the exported performance metrics, in particular the mapping from PMID to external name, and vice-versa.

1.1.3. Distributed PMNS

As of release 2.0 release, PMNS operations by default are directed to the host or archive that is the source of the desired performance metrics.

In Figure 1-2, both Performance Metrics Collection Daemon (PMCD) processes would respond to PMNS queries from monitoring tools by referring to their local PMNS. If different PMDAs were installed on the two hosts, then the PMNS used by each PMCD would be different, to reflect variations in available metrics on the two hosts.

Distributed PMNS services necessitated changes to PCP protocols between client applications and PMCD, and to the internal format of PCP archive files.

The -n pmnsfile option may be used with all PCP monitoring tools to force use of the local PMNS in preference to the PMNS at the source of the metrics.

1.1.4. Retrospective Sources of Performance Metrics

The distributed collection architecture described in the previous section is used when PMAPI clients are requesting performance metrics from a real-time or live source.

The PMAPI also supports delivery of performance metrics from a historical source in the form of a PCP archive log. Archive logs are created using the pmlogger utility, and are replayed in an architecture as shown in Figure 1-3.

Figure 1-3. Architecture for Retrospective Analysis

Architecture for Retrospective Analysis

1.2. Overview of Component Software

Performance Co-Pilot (PCP) is composed of text-based tools, optional graphical tools, and related commands. Each tool or command is fully documented by a man page. These man pages are named after the tools or commands they describe, and are accessible through the man command. For example, to see the pminfo(1) man page for the pminfo command, enter this command:

man pminfo

Many PCP tools and commands are accessible from an Icon Catalog on the IRIX desktop, grouped under PerfTools. In the Toolchest Find menu, choose PerfTools; an Icon Catalog appears, containing clickable PCP programs. To bring up a Web-based introduction to Performance Co-Pilot, click the AboutPCP icon.

A list of PCP tools and commands, grouped by functionality, is provided in the following sections.

1.2.1. Application and Agent Development

The following PCP tools aid the development of new programs to consume performance data, and new agents to export performance data within the PCP framework:


Checks the consistency of performance metrics help database files.


Allows PMDA behavior to be exercised and tested. It is an interactive debugger for PMDAs.


Generates the database files for one or more source files of PCP help text.


Defines a procedural interface for developing PCP client applications. It is the Performance Metrics Application Programming Interface (PMAPI).


Is a simple client that uses the PMAPI to report some high-level system performance metrics. The source code for pmclient is included in the distribution.


Is a library used by many shipped PMDAs to communicate with a pmcd process. It can expedite the development of new and custom PMDAs.


Generates C declarations and cpp macros to aid the development of customized programs that use the facilities of PCP. It is a program development tool.

1.3. PMDA Development

A collection of Performance Metrics Domain Agents (PMDAs) are provided with PCP to extract performance metrics. Each PMDA encapsulates domain-specific knowledge and methods about performance metrics that implement the uniform access protocols and functional semantics of the PCP. There is one PMDA for the operating system, another for process specific statistics, one each for common DBMS products, and so on. Thus, the range of performance metrics can be easily extended by implementing and integrating new PMDAs. Chapter 2, “Writing a PMDA”, is a step-by-step guide to writing your own PMDA.

1.3.1. Overview

Once you are familiar with the PCP and PMDA frameworks, you can quickly implement a new PMDA with only a few data structures and functions. This book contains detailed discussions of PMDA architecture and the integration of PMDAs into the PCP framework. This includes integration with PMCD. However, details of extracting performance metrics from the underlying instrumentation vary from one domain to another and are not covered in this book.

A PMDA is responsible for a set of performance metrics, in the sense that it must respond to requests from PMCD for information about performance metrics, instance domains, and instantiated values. The PMCD process generates requests on behalf of monitoring tools that make requests using PMAPI functions.

You can incorporate new performance metrics into the PCP framework by creating a PMDA, then reconfiguring PMCD to communicate with the new PMDA.

1.3.2. Building a PMDA

A PMDA interacts with PMCD across one of several well-defined interfaces and protocol mechanisms. These implementation options are described in the Performance Co-Pilot for IRIX Advanced User's and Administrator's Guide or the Performance Co-Pilot for IA-64 Linux User's and Administrator's Guide.

It is strongly recommended that code for a new PMDA be based on the source of one of the demonstration PMDAs below the /var/pcp/pmdas directory. In-Process (DSO) Method

This method of building a PMDA uses a Dynamic Shared Object (DSO) that is attached by PMCD, using dlopen, at initialization time. This is the highest performance option (there is no context switching and no interprocess communication (IPC) between the PMCD and the PMDA), but is operationally intractable in some situations. For example, difficulties arise where special access permissions are required to read the instrumentation behind the performance metrics, or where the performance metrics are provided by an existing process with a different protocol interface. The DSO PMDA effectively executes as part of PMCD; so care is required when crafting a PMDA in this manner.

When developing PMDAs for IRIX or Linux, multiple object code formats for the DSO may be required because PMCD must execute with the same object code format as the running operating system kernel. This would be o32 for some low-end platforms (IRIX 6.3 and earlier), n32 for other low-end platforms (IRIX 6.5 and later), and n64 for high-end platforms. Daemon Process Method

Functionally, this method may be thought of as a DSO implementation with a standard main routine conversion wrapper so that communication with PMCD uses message passing rather than direct procedure calls. (See /var/pcp/pmdas/trivial/trivial.c file.)

The daemon PMDA is actually the most common, because it allows multiple threads of control, permits linking with existing dynamic libraries, and provides more resilient error encapsulation than the DSO method. Shell Process Method

This method offers the least performance, but may be well-suited for rapid prototyping of performance metrics, or for diagnostic metrics that are not going into production.

Implementation of the ASCII protocols is rather lengthy. The suggested approach is to take the /var/pcp/pmdas/news/pmdanews PMDA as an illustrative example, and adapt it for the particular metrics of interest.

Note: The ASCII protocols have not been extensively used; so their support may be discontinued in a future PCP release. Newer versions of the PMDA libraries have dramatically reduced the code development effort required for a new PMDA (either the DSO or daemon approach), thereby reducing the need for ASCII protocols.

1.4. Client Development and PMAPI

Application developers are encouraged to create new PCP client applications to monitor, display, and analyze performance data in a manner suited to their particular site, application suite, or information processing environment.

PCP client applications are programmed using the Performance Metrics Application Programming Interface (PMAPI), documented in Chapter 3, “PMAPI--The Performance Metrics API”. The PMAPI, which provides performance tool developers with access to all of the distributed services of the Performance Metrics Collection Subsystem (PMCS), is the interface used by the standard PCP utilities.

Source for a sample IRIX PMAPI client may be found in the /var/pcp/demos/pmclient directory if the pcp.sw.demo subsystem has been installed. Source for a sample Linux PMAPI client may be found in the /usr/share/pcp/demos/pmclient directory.

1.5. Library Reentrancy and Threaded Applications

Most of the PCP libraries are not thread safe. This is a deliberate design decision to trade-off commonly required performance and efficiency against the less common requirement for multiple threads of control to call the PCP libraries.

The simplest and safest programming model is to designate at most one thread to make calls into the PCP libraries. This approach applies to both PMDAs using libpcp_pmda and monitoring applications using PMAPI and calling the libpcp library.

An important exception is the libpcp_trace library for instrumenting applications; it is thread safe.

Particular care should be taken with the utility functions in the libpcp library; for example, pmprintf and pmflush share a buffer that may be corrupted if calls to these functions from multiple threads are overlapped.