Performance Co-Pilot (PCP) provides a systems-level suite of tools that cooperate to deliver distributed, integrated performance management services. PCP is designed for the in-depth analysis and sophisticated control that are needed to understand and manage the hardest performance problems in our most complex systems.
PCP provides unparalleled power to quickly isolate and understand performance behavior, resource utilization, activity levels and performance bottlenecks.
Performance data may be collected and exported from multiple sources, most notably the hardware platform, the operating system kernel, layered services, and end-user applications.
There are several ways to extend PCP by programming certain of its components:
By writing a Performance Metrics Domain Agent (PMDA) to collect performance metrics from an uncharted performance domain (Chapter 2, “Writing a PMDA”)
By creating new analysis or visualization tools using documented functions from the Performance Metrics Application Programming Interface (PMAPI) (Chapter 3, “PMAPI--The Performance Metrics API”)
By adding performance instrumentation to an application using the trace facilities of the PCP trace library (libpcp_trace) and the trace PMDA (Chapter 4, “Trace PMDA”)
In addition, the topic of customizing a PCP installation is covered in the chapter on customizing and extending PCP service in the Performance Co-Pilot for IRIX Advanced User's and Administrator's Guide or in the Performance Co-Pilot for IA-64 Linux User's and Administrator's Guide.
This section gives a brief overview of PCP architecture. For an explanation of terms and acronyms, refer to Appendix A, “Acronyms”.
PCP consists of several monitoring and collecting tools. Monitoring tools such as pmval and pminfo visualize metrics, but have minimal interaction with target systems. Collection tools, called PMDAs, extract performance values from target systems, but do not provide graphical user interfaces.
Systems supporting PCP services are broadly classified into two categories:
| Collector | Hosts that have the PMCD and one or more PMDAs running to collect and export performance metrics | |
| Monitor | Hosts that import performance metrics from one or more collector hosts to be consumed by tools to monitor, manage, or record the performance of the collector hosts |
Each PCP enabled host can operate as a collector, or a monitor, or both.
There are separate node-locked licenses for additional tools when available on IRIX systems for collector and monitor functions.
Figure 1-1 shows the architecture of PCP. The monitoring tools consume and process performance data using a public interface, the Performance Metrics Application Programming Interface (PMAPI).
Below the PMAPI level is the PMCD process, which acts in a coordinating role, accepting requests from clients, routing requests to one or more PMDAs, aggregating responses from the PMDAs, and responding to the requesting client.
Each performance metric domain (such as IRIX, Linux, or some Database Management System (DBMS)) has a well-defined name space for referring to the specific performance metrics it knows how to collect.
The performance metrics collection architecture is distributed, in the sense that any monitoring tool may be executing remotely. However, a PMDA is expected to be running on the operating system for which it is collecting performance measurements; there are some notable PMDAs such as Cisco and Array that are exceptions, and collect performance data from remote systems.
As shown in Figure 1-2, monitoring tools communicate only with PMCD. The PMDAs are controlled by PMCD and respond to requests from the monitoring tools that are forwarded by PMCD to the relevant PMDAs on the colllector host.
The host running the monitoring tools does not require any collection tools, including PMCD, since all requests for metrics are sent to the PMCD process on the collector host.
The connections between monitoring tools and PMCD processes are managed in libpcp, below the PMAPI level; see the pmapi(3) man page. Connections between PMDAs and PMCD are managed by the PMDA functions; see the pmda(3) and pmcd(1) man pages. There can be multiple monitor clients and multiple PMDAs on the one host, but there may be only one PMCD process.
Each PMDA provides a domain of metrics, whether they be for the operating system, a database manager, a layered service, or an application module. These metrics are referred to by name inside the user interface, and with a numeric Performance Metric Identifier (PMID) within the underlying PMAPI.
The PMID consists of three fields: the domain, the cluster, and the item number of the metric. The domain is a unique number assigned to each PMDA. For example, two metrics with the same domain number must be from the same PMDA. The cluster and item numbers allow metrics to be easily organized into groups within the PMDA, and provide a hierarchical taxonomy to guarantee uniqueness within each PMDA.
The Performance Metrics Name Space (PMNS) describes the exported performance metrics, in particular the mapping from PMID to external name, and vice-versa.
As of release 2.0 release, PMNS operations by default are directed to the host or archive that is the source of the desired performance metrics.
In Figure 1-2, both Performance Metrics Collection Daemon (PMCD) processes would respond to PMNS queries from monitoring tools by referring to their local PMNS. If different PMDAs were installed on the two hosts, then the PMNS used by each PMCD would be different, to reflect variations in available metrics on the two hosts.
Distributed PMNS services necessitated changes to PCP protocols between client applications and PMCD, and to the internal format of PCP archive files.
The -n pmnsfile option may be used with all PCP monitoring tools to force use of the local PMNS in preference to the PMNS at the source of the metrics.
The distributed collection architecture described in the previous section is used when PMAPI clients are requesting performance metrics from a real-time or live source.
The PMAPI also supports delivery of performance metrics from a historical source in the form of a PCP archive log. Archive logs are created using the pmlogger utility, and are replayed in an architecture as shown in Figure 1-3.
Performance Co-Pilot (PCP) is composed of text-based tools, optional graphical tools, and related commands. Each tool or command is fully documented by a man page. These man pages are named after the tools or commands they describe, and are accessible through the man command. For example, to see the pminfo(1) man page for the pminfo command, enter this command:
man pminfo |
Many PCP tools and commands are accessible from an Icon Catalog on the IRIX desktop, grouped under PerfTools. In the Toolchest Find menu, choose PerfTools; an Icon Catalog appears, containing clickable PCP programs. To bring up a Web-based introduction to Performance Co-Pilot, click the AboutPCP icon.
A list of PCP tools and commands, grouped by functionality, is provided in the following sections.
The following PCP tools aid the development of new programs to consume performance data, and new agents to export performance data within the PCP framework:
A collection of Performance Metrics Domain Agents (PMDAs) are provided with PCP to extract performance metrics. Each PMDA encapsulates domain-specific knowledge and methods about performance metrics that implement the uniform access protocols and functional semantics of the PCP. There is one PMDA for the operating system, another for process specific statistics, one each for common DBMS products, and so on. Thus, the range of performance metrics can be easily extended by implementing and integrating new PMDAs. Chapter 2, “Writing a PMDA”, is a step-by-step guide to writing your own PMDA.
Once you are familiar with the PCP and PMDA frameworks, you can quickly implement a new PMDA with only a few data structures and functions. This book contains detailed discussions of PMDA architecture and the integration of PMDAs into the PCP framework. This includes integration with PMCD. However, details of extracting performance metrics from the underlying instrumentation vary from one domain to another and are not covered in this book.
A PMDA is responsible for a set of performance metrics, in the sense that it must respond to requests from PMCD for information about performance metrics, instance domains, and instantiated values. The PMCD process generates requests on behalf of monitoring tools that make requests using PMAPI functions.
You can incorporate new performance metrics into the PCP framework by creating a PMDA, then reconfiguring PMCD to communicate with the new PMDA.
A PMDA interacts with PMCD across one of several well-defined interfaces and protocol mechanisms. These implementation options are described in the Performance Co-Pilot for IRIX Advanced User's and Administrator's Guide or the Performance Co-Pilot for IA-64 Linux User's and Administrator's Guide.
It is strongly recommended that code for a new PMDA be based on the source of one of the demonstration PMDAs below the /var/pcp/pmdas directory.
This method of building a PMDA uses a Dynamic Shared Object (DSO) that is attached by PMCD, using dlopen, at initialization time. This is the highest performance option (there is no context switching and no interprocess communication (IPC) between the PMCD and the PMDA), but is operationally intractable in some situations. For example, difficulties arise where special access permissions are required to read the instrumentation behind the performance metrics, or where the performance metrics are provided by an existing process with a different protocol interface. The DSO PMDA effectively executes as part of PMCD; so care is required when crafting a PMDA in this manner.
When developing PMDAs for IRIX or Linux, multiple object code formats for the DSO may be required because PMCD must execute with the same object code format as the running operating system kernel. This would be o32 for some low-end platforms (IRIX 6.3 and earlier), n32 for other low-end platforms (IRIX 6.5 and later), and n64 for high-end platforms.
Functionally, this method may be thought of as a DSO implementation with a standard main routine conversion wrapper so that communication with PMCD uses message passing rather than direct procedure calls. (See /var/pcp/pmdas/trivial/trivial.c file.)
The daemon PMDA is actually the most common, because it allows multiple threads of control, permits linking with existing dynamic libraries, and provides more resilient error encapsulation than the DSO method.
This method offers the least performance, but may be well-suited for rapid prototyping of performance metrics, or for diagnostic metrics that are not going into production.
Implementation of the ASCII protocols is rather lengthy. The suggested approach is to take the /var/pcp/pmdas/news/pmdanews PMDA as an illustrative example, and adapt it for the particular metrics of interest.
 | Note: The ASCII protocols have not been extensively used; so their support may be discontinued in a future PCP release. Newer versions of the PMDA libraries have dramatically reduced the code development effort required for a new PMDA (either the DSO or daemon approach), thereby reducing the need for ASCII protocols. |
Application developers are encouraged to create new PCP client applications to monitor, display, and analyze performance data in a manner suited to their particular site, application suite, or information processing environment.
PCP client applications are programmed using the Performance Metrics Application Programming Interface (PMAPI), documented in Chapter 3, “PMAPI--The Performance Metrics API”. The PMAPI, which provides performance tool developers with access to all of the distributed services of the Performance Metrics Collection Subsystem (PMCS), is the interface used by the standard PCP utilities.
Source for a sample IRIX PMAPI client may be found in the /var/pcp/demos/pmclient directory if the pcp.sw.demo subsystem has been installed. Source for a sample Linux PMAPI client may be found in the /usr/share/pcp/demos/pmclient directory.
Most of the PCP libraries are not thread safe. This is a deliberate design decision to trade-off commonly required performance and efficiency against the less common requirement for multiple threads of control to call the PCP libraries.
The simplest and safest programming model is to designate at most one thread to make calls into the PCP libraries. This approach applies to both PMDAs using libpcp_pmda and monitoring applications using PMAPI and calling the libpcp library.
An important exception is the libpcp_trace library for instrumenting applications; it is thread safe.
Particular care should be taken with the utility functions in the libpcp library; for example, pmprintf and pmflush share a buffer that may be corrupted if calls to these functions from multiple threads are overlapped.