Chapter 2. Writing a PMDA

This chapter constitutes a programmer's guide to writing a Performance Metrics Domain Agent (PMDA) for Performance Co-Pilot (PCP).

The presentation assumes the developer is using the standard PCP libpcp_pmda library, as documented in the pmda(3) and associated man pages.

2.1. Implementing a PMDA

The job of a PMDA is to gather performance data and report them to the Performance Metrics Collection Daemon (PMCD) in response to requests from PCP monitoring tools routed to the PMDA via PMCD.

An important requirement for any PMDA is that it have low latency response to requests from PMCD. Either the PMDA must use a quick access method and a single thread of control, or it must have asynchronous refresh and two threads of control: one for communicating with PMCD, the other for updating the performance data.

The PMDA is typically acting as a gateway between the target domain (that is, the performance instrumentation in an application program or service) and the PCP framework. The PMDA may extract the information using one of a number of possible export options that include a shared memory segment or mmap file; a sequential log file (where the PMDA parses the tail of the log file to extract the information); a snapshot file (the PMDA rereads the file as required); or application-specific communication services (IPC). The choice of export methodology is typically determined by the source of the instrumentation (the target domain) rather than by the PMDA.

Procedure 2-1 describes the suggested steps for designing and implementing a PMDA:

Procedure 2-1. Creating a PMDA

  1. Determine how to extract the metrics from the target domain.

  2. Select an appropriate architecture for the PMDA (daemon or DSO, IPC, sproc).

  3. Define the metrics and instances that the PMDA will support.

  4. Implement the functionality to extract the metric values.

  5. Assign Performance Metric Identifiers (PMIDs) for the metrics, along with names for the metrics in the Performance Metrics Name Space (PMNS).

  6. Specify the help file and control data structures for metrics and instances that are required by the standard PMDA implementation library functions.

  7. Write code to supply the metrics and associated information to PMCD.

  8. Implement any PMDA-specific callbacks, and PMDA initialization functions.

  9. Exercise and test the PMDA with the purpose-built PMDA debugger; see the dbpmda(1) man page.

  10. Install and connect the PMDA to a running PMCD process; see the pmcd(1) man page.

  11. Configure or develop tools to use the new metrics. For examples of visualization tools, see the pmchart(1), pmgadgets(1), and pmview(1) man pages (IRIX only). For an examples of text-based tools, see the pmval(1) and pminfo(1) man pages. For examples of alarm tools, see the pmie(1) and pmieconf(1) man pages.

    Where appropriate, define pmlogger configurations suitable for creating PCP archives containing the new metrics. For more information, see the pmlogger(1) man page.

    2.2. PMDA Architecture

    This section discusses the two methods of connecting a PMDA to a PMCD process:

    • As a separate process using some interprocess communication (IPC) protocol.

    • As a dynamically attached library (that is, a dynamic shared object or DSO).

    2.2.1. Overview

    All PMDAs are launched and controlled by the PMCD process on the local host. PMCD receives requests from the monitoring tools and forwards them to the PMDAs. Responses, when required, are returned through PMCD to the clients. The requests fall into a small number of categories, and the PMDA must handle each request type. For a DSO PMDA, each request type corresponds to a method in the agent. For a daemon PMDA, each request translates to a message or protocol data unit (PDU) that may be sent to a PMDA from PMCD.

    For daemon PMDA, the following request PDUs must be supported:


    Request for metric values (see the pmFetch(3) man page.)


    A list of instances required for the corresponding metrics in subsequent fetches (see the pmAddProfile(3) man page).


    Request for a particular instance domain for instance descriptions (see the pmGetInDom(3) man page).


    Request for metadata describing metrics (see the pmLookupDesc(3) man page).


    Request for metric help text (see the pmLookupText(3) man page).


    Values to store into metrics (see the pmStore(3) man page).

    Each PMDA is associated with a unique domain number that is encoded in the domain field of metric and instance identifiers, and PMCD uses the domain number to determine which PMDA can handle the components of any given client request.

    2.2.2. DSO PMDA

    Each PMDA is required to implement a function that handles each of the request types. By implementing these functions as library functions, a PMDA can be implemented as a dynamically shared object (DSO) and attached by PMCD at run time with the dlopen call; see the dlopen(3) man page. This eliminates the need for an IPC layer (typically a UNIX pipe) between each PMDA and PMCD, because each request becomes a function call rather than a message exchange. The required library functions are detailed in Section 2.5.

    A PMDA that interacts with PMCD in this fashion must abide by a formal initialization protocol so that PMCD can discover the location of the library functions that are subsequently called with function pointers. When a DSO PMDA is installed, the PMCD configuration file, /etc/pmcd.conf on IRIX; /var/pcp/config/pmcd/pmcd.conf, is updated to reflect the domain and name of the PMDA, the location of the shared object, and the name of the initialization function. The initialization sequence is discussed in Section 2.6.

    As superuser, install the simple PMDA as a DSO, as shown in Example 2-1 and Example 2-2, and observe the changes in the PMCD configuration file. The output may differ slightly depending on the other PMDAs you have installed.

    Example 2-1. Simple IRIX PMDA as a DSO

    # cd /var/pcp/pmdas/simple
    # cat /etc/pmcd.conf
    # Name  Id      IPC     IPC Params      File/Cmd 
    irix    1       dso     irix_init 
    pmcd    2       dso     pmcd_init 
    proc    3       dso     proc_init 
    # ./Install
    You will need to choose an appropriate configuration for installation 
    of the “simple” Performance Metrics Domain Agent (PMDA).
    collector   collect performance statistics on this system
      monitor     allow this system to monitor local and/or remote systems
      both         collector and monitor configuration for this system
    Please enter c(ollector) or m(onitor) or b(oth) [b] both
    Updating the Performance Metrics Name Space (PMNS) ... 
    Installing pmchart view(s) ...
    Install simple as a daemon or dso agent? [daemon] dso
    Check simple metrics have appeared ... 5 metrics and 9 values
    # cat /etc/pmcd.conf
    # Name  Id      IPC     IPC Params      File/Cmd 
    irix    1       dso     irix_init 
    pmcd    2       dso     pmcd_init 
    proc    3       dso     proc_init 
    simple  253     dso     simple_init

    As can be seen from the contents of /etc/pmcd.conf, the DSO version of the simple PMDA is in a library named and has an initialization function called simple_init. The domain of the simple PMDA is 253, as shown in the column headed Id.

    Example 2-2. Simple Linux PMDA as a DSO

    cat /var/pcp/config/pmcd/pmcd.conf
    # Performance Metrics Domain Specifications
    # This file is automatically generated during the build
    # Name  Id      IPC     IPC Params      File/Cmd
    pmcd    2       dso     pmcd_init       /var/pcp/pmdas/pmcd/
    linux   60      dso     linux_init      /var/pcp/pmdas/linux/
    snia 63 pipe binary /var/pcp/pmdas/snia/pmdasnia -d 63
    simple  254     dso     simple_init

    As can be seen from the contents of /var/pcp/config/pmcd/pmcd.conf, the DSO version of the simple PMDA is in a library named and has an initialization function called simple_init. The domain of the simple PMDA is 254, as shown in the column headed Id.

    2.2.3. Daemon PMDA

    A DSO PMDA provides the most efficient communication between the PMDA and PMCD. This approach has some disadvantages resulting from the DSO PMDA being the same process as PMCD:

    • An error or bug that causes a DSO PMDA to exit also causes PMCD to exit.

    • There is only one thread of control in PMCD; as a result, a computationally expensive PMDA, or worse, a PMDA that blocks for I/O, adversely affects the performance of PMCD.

    • The PMCD runs as superuser; so any DSO PMDAs also run as superuser.

    • A memory leak in a DSO PMDA also causes a memory leak for PMCD.

    Consequently, many PMDAs are implemented as a daemon process.

    The libpcp_pmda library is designed to allow simple implementation of a PMDA that runs as a separate process. The library functions provide a message passing layer acting as a generic wrapper that accepts PDUs, makes library calls using the standard DSO PMDA interface, and sends PDUs. Therefore, you can implement a PMDA as a DSO and then install it as either a daemon or a DSO, depending on the presence or absence of the generic wrapper.

    The PMCD process launches a daemon PMDA with fork and execv. You can easily connect a pipe to the PMDA using standard input and output. The PMCD process may also connect to a daemon PMDA using TCP/IP or UNIX domain sockets; see the inet(7) or unix(7) man page.

    As superuser, install the simple PMDA as a daemon process as shown in Example 2-3 for IRIX and Example 2-4 for Linux. As in Example 2-1, the output may differ due to other PMDAs already installed.

    Example 2-3. Simple IRIX PMDA as a Daemon

    The specification for the simple PMDA now states the connection type of pipe to PMCD and the executable image for the PMDA is /var/pcp/pmdas/simple/pmdasimple, using domain number 253.

    # cd /var/pcp/pmdas/simple
    # ./Install 
    Install simple as a daemon or dso agent? [daemon] daemon 
    PMCD should communicate with the daemon via pipe or socket? [pipe] pipe
    # cat /etc/pmcd.conf 
    # Name  Id   IPC     IPC Params File/Cmd 
    irix    1    dso     irix_init 
    pmcd    2    dso     pmcd_init 
    proc    3    dso     proc_init 
    simple  253  pipe    binary     /var/pcp/pmdas/simple/pmdasimple -d 253

    Example 2-4. Simple Linux PMDA as a Daemon

    The specification for the simple PMDA now states the connection type of pipe to PMCD and the executable image for the PMDA is /var/pcp/pmdas/simple/pmdasimple, using domain number 253.

    # cd /var/pcp/pmdas/simple
    # ./Install
    Install simple as a daemon or dso agent? [daemon] daemon 
    PMCD should communicate with the daemon via pipe or socket? [pipe] pipe
    # cat /var/pcp/config/pmcd/pmcd.conf 
    # Performance Metrics Domain Specifications
    # This file is automatically generated during the build
    # Name  Id      IPC     IPC Params      File/Cmd
    pmcd    2       dso     pmcd_init       /var/pcp/pmdas/pmcd/
    linux   60      dso     linux_init      /var/pcp/pmdas/linux/
    snia 63 pipe binary /var/pcp/pmdas/snia/pmdasnia -d 63 
    simple  253  pipe    binary     /var/pcp/pmdas/simple/pmdasimple -d 253

    2.2.4. Caching PMDA

    When either the cost or latency associated with collecting performance metrics is high, the PMDA implementer may choose to trade off the currency of the performance data to reduce the PMDA resource demands or the fetch latency time.

    One scheme for doing this is called a caching PMDA, which periodically instantiates values for the performance metrics and responds to each request from PMCD with the most recently instantiated (or cached) values, as opposed to instantiating current values on demand when the PMCD asks for them.

    The Cisco PMDA is an example of a caching PMDA. For additional information, see the contents of the /var/pcp/pmdas/cisco directory and the pmdacisco(1) man page.

    2.3. Domains, Metrics, and Instances

    This section defines metrics and instances, discusses how they should be designed for a particular target domain, and shows how to implement support for them.

    The examples in this section are drawn from the trivial and simple PMDAs, which are distributed in source format with PCP. Refer to the /var/pcp/pmdas/trivial and /var/pcp/pmdas/simple directories, respectively.

    2.3.1. Overview

    Domains are autonomous performance areas, such as the operating system or a layered service or a particular application. Metrics are raw performance data for a domain, and typically quantify activity levels, resource utilization or quality of service. Instances are sets of related metrics, as for multiple processors, or multiple service classes, or multiple transaction types.

    PCP employs the following simple and uniform data model to accommodate the demands of performance metrics drawn from multiple domains:

    • Each metric has an identifier that is unique across all metrics for all PMDAs on a particular host.

    • Externally, metrics are assigned names for user convenience--typically there is a 1:1 relationship between a metric name and a metric identifier.

    • The PMDA implementation determines if a particular metric has a singular value or a set of (zero or more) values. For instance, the metric hinv.ndisk counts the number of disks and has only one value on a host, whereas the metric counts disk I/O operations and has one value for each disk on the host.

    • If a metric has a set of values, then members of the set are differentiated by instances. The set of instances associated with a metric is an instance domain. For example, the set of metrics is defined over an instance domain that has one member per disk spindle.

    The selection of metrics and instances is an important design decision for a PMDA implementer. The metrics and instances for a target domain should have the following qualities:

    • Obvious to a user

    • Consistent across the domain

    • Accurately representative of the operational and functional aspects of the domain

    For each metric, you should also consider these questions:

    • How useful is this value?

    • What units give a good sense of scale?

    • What name gives a good description of the metric's meaning?

    • Can this metric be combined with another to convey the same useful information?

    As with all programming tasks, expect to refine the choice of metrics and instances several times during the development of the PMDA.

    2.3.2. Domains

    Each PMDA must be uniquely identified by PMCD so that requests from clients can be efficiently routed to the appropriate PMDA. The unique identifier, the PMDA's domain, is encoded within the metrics and instance domain identifiers so that they are associated with the correct PMDA, and so that they are unique, regardless of the number of PMDAs that are connected to the PMCD process.

    The default domain number for each PMDA is defined in /var/pcp/pmns/stdpmid. This file is a simple table of PMDA names and their corresponding domain number. However, a PMDA does not have to use this domain number--the file is only a guide to help avoid domain number clashes when PMDAs are installed and activated.

    The domain number a PMDA uses is passed to the PMDA by PMCD when the PMDA is launched. Therefore, any data structures that require the PMDA's domain number must be set up when the PMDA is initialized, rather than declared statically. The protocol for PMDA initialization provides a standard way for a PMDA to implement this run-time initialization.

    Tip: Although uniqueness of the domain number in the /etc/pmcd.conf control file used by PMCD is all that is required for successful starting of PMCD and the associated PMDAs, the developer of a new PMDA is encouraged to add the default domain number for each new PMDA to the /var/pcp/pmns/stdpmid.local file and then to run the Make.stdpmid script in /var/pcp/pmns to recreate /var/pcp/pmns/stdpmid; this file acts as a repository for documenting the known default domain numbers.

    2.3.3. Metrics

    A PMDA provides support for a collection of metrics. In addition to the obvious performance metrics, and the measures of time, activity and resource utilization, the metrics should also describe how the target domain has been configured, as this can greatly affect the correct interpretation of the observed performance. For example, metrics that describe network transfer rates should also describe the number and type of network interfaces connected to the host.

    In addition, the metrics should describe how the PMDA has been configured. For example, if the PMDA was periodically probing a system to measure quality of service, there should be metrics for the delay between probes, the number of probes attempted, plus probe success and failure counters. It may also be appropriate to allow values to be stored (see the pmstore(1) man page) into the delay metric, so that the delay used by the PMDA can be altered dynamically. Data Structures

    Each metric must be described in a pmDesc structure; see the pmLookupDesc(3) man page:

    typedef struct { 
        pmID        pmid;           /* unique identifier */ 
        int         type;           /* base data type */ 
        pmInDom     indom;          /* instance domain */ 
        int         sem;            /* semantics of value */ 
        pmUnits     units;          /* dimension and units */ 
    } pmDesc;

    This structure contains the following fields:


    A unique identifier, Performance Metric Identifier (PMID), that differentiates this metric from other metrics across the union of all PMDAs


    A data type indicator showing whether the format is an integer (32 or 64 bit, signed or unsigned); float; double; string; or arbitrary aggregate of binary data


    An instance domain identifier that links this metric to an instance domain


    An encoding of the value's semantics (counter, instantaneous, or discrete)


    A description of the value's units based on dimension and scale in the three orthogonal dimensions of space, time, and count (or events)

    Symbolic constants of the form PM_TYPE_*, PM_SEM_*, PM_SPACE_*, PM_TIME_*, and PM_COUNT_* are defined in the /usr/include/pcp/pmapi.h file. You may use them to initialize the elements of a pmDesc structure. The pmID type is an unsigned integer that can be safely cast to a _pmID_int structure, which contains fields defining the metric's (PMDA's) domain, cluster, and item number as shown in Example 2-5 for IRIX and Example 2-6 for Linux:

    Example 2-5. _pmID_int Structure in IRIX

    typedef struct { 
            int             pad:2; 
            unsigned int    domain:8; 
            unsigned int    cluster:12; 
            unsigned int    item:10; 
    } _pmID_int;

    Example 2-6. _pmID_int Structure in Linux

    typedef struct { 
            unsigned int    item:10; 
            unsigned int    cluster:12; 
            unsigned int    domain:8; 
            int             pad:2; 
    } _pmID_int;

    For additional information, see the /usr/include/pcp/impl.h file.

    The pad field should be ignored. The domain number should be set at run time when the PMDA is initialized. The PMDA_PMID macro defined in /usr/include/pcp/pmapi.h can be used to set the cluster and item fields at compile time, as these should always be known and fixed for a particular metric.

    Note: The three components of the PMID should correspond exactly to the three-part definition of the PMID for the corresponding metric in the PMNS described in Section 2.4.3.

    A table of pmdaMetric structures should be defined within the PMDA, with one structure per metric as shown in Example 2-7.

    Example 2-7. pmdaMetric Structure

    typedef struct { 
        void        *m_user;        /* for users external use */ 
        pmDesc      m_desc;         /* metric description */ 
    } pmdaMetric;

    This structure contains a pmDesc structure and a handle that allows PMDA-specific structures to be associated with each metric. For example, m_user could be a pointer to a global variable containing the metric value, or a pointer to a function that may be called to instantiate the metric's value.

    The trivial PMDA, shown in Example 2-8, has only a singular metric (that is, no instance domains):

    Example 2-8. Trivial PMDA

    static pmdaMetric metrictab[] = {
    /* time */ 
     { (void *)0,  
         PMDA_PMUNITS (0, 1, 0, 0, PM_TIME_SEC, 0) ), ) 

    This single metric (trivial.time) has the following:

    • A PMID with a cluster of 0 and an item of 1

    • An unsigned 32-bit integer (PM_TYPE_U32 )

    • A singular value and hence no instance domain (PM_INDOM_NULL)

    • An instantaneous semantic value (PM_SEM_INSTANT)

    • Dimension “time” and the units “seconds” Semantics

    The metric's semantics describe how PCP tools should interpret the metric's value. The following are the possible semantic types:

    • Counter ( PM_SEM_COUNTER)

    • Instantaneous value (PM_SEM_INSTANT)

    • Discrete value (PM_SEM_DISCRETE)

    A counter should be a value that monotonically increases (or monotonically decreases, which is less likely) with respect to time, so that the rate of change should be used in preference to the actual value. Rate conversion is not appropriate for metrics with instantaneous values, as the value is a snapshot and there is no basis for assuming any values that might have been observed between snapshots. Discrete is similar to instantaneous; however, once observed it is presumed the value will persist for an extended period (for example, system configuration, static tuning parameters and most metrics with nonnumeric values).

    For a given time interval covering six consecutive timestamps, each spanning two units of time, themetric values, in Example 2-9, are exported from a PMDA (“N/A” implies no value is available):

    Example 2-9. Effect of Semantics on a Metric

    Timestamps:         1   3   5   7   9  11 
    Value:             10  30  60  80  90 N/A

    The default display of the values would be as follows:

    Timestamps:         1   3   5   7   9  11 
    Counter           N/A  10  15  10   5 N/A 
    Instantaneous      10  30  60  80  90 N/A 
    Discrete           10  30  60  80  90  90

    2.3.4. Instances

    Singular metrics have only one value and no associated instance domain. Some metrics contain a set of values that share a common set of semantics for a specific instance, such as one value per processor, or one value per disk spindle, and so on.

    Note: The PMDA implementation is solely responsible for choosing the instance identifiers that differentiate instances within the instance domain. The PMDA is also responsible for ensuring the uniqueness of instance identifiers in any instance domain. N Dimensional Data

    Where the performance data can be represented as scalar values (singular metrics) or one-dimensional arrays or lists (metrics with an instance domain), the PCP framework is more than adequate. In the case of metrics with an instance domain, each array or list element is associated with an instance from the instance domain.

    To represent two or more dimensional arrays, the coordinates must be one of the following:

    • Mapped onto one dimensional coordinates.

    • Enumerated into the Performance Metrics Name Space (PMNS).

    For example, this 2 x 3 array of values called M can be represented as instances 1,..., 6 for a metric M:

      M[1]   M[2]   M[3] 
      M[4]   M[5]   M[6]

    Or they can be represented as instances 1, 2, 3 for metric M1 and instances 1, 2, 3 for metric M2:

      M1[1]  M1[2]  M1[3] 
      M2[1]  M2[2]  M2[3]

    The PMDA implementer must decide and consistently export this encoding from the N-dimensional instrumentation to the 1-dimensional data model of the PCP.

    In certain special cases (for example, such as for a histogram), it may be appropriate to export an array of values as raw binary data (the type encoding in the descriptor is PM_TYPE_AGGREGATE). However, this requires the development of special PMAPI client tools, because the standard PCP tools have no knowledge of the structure and interpretation of the binary data. Data Structures

    If the PMDA is required to support instance domains, then for each instance domain the unique internal instance identifier and external instance identifier should be defined using a pmdaInstid structure as shown in Example 2-10:

    Example 2-10. pmdaInstid Structure

    typedef struct { 
        int         i_inst;         /* internal instance identifier */ 
        char        *i_name;        /* external instance identifier */ 
    } pmdaInstid;

    The i_inst instance identifier must be a unique integer within a particular instance domain.

    The complete instance domain description is specified in a pmdaIndom structure as shown in Example 2-11:

    Example 2-11. pmdaIndom Structure

    typedef struct { 
        pmInDom     it_indom;       /* indom, filled in */ 
        int         it_numinst;     /* number of instances */ 
        pmdaInstid  *it_set;        /* instance identifiers */ 
    } pmdaIndom;

    The it_indom element contains a pmInDom that must be unique across every PMDA. The other fields of the pmdaIndom structure are the number of instances in the instance domain and a pointer to an array of instance descriptions.

    Example 2-12 for IRIX and Example 2-13 for Linux shows that the pmInDom can be safely cast to _pmInDom_int, which specifies the PMDA's domain and the instance number within the PMDA:

    Example 2-12. _pmInDom_int Structure in IRIX

    typedef struct { 
            int             pad:2; 
            unsigned int    domain:8;   /* the administrative PMD */ 
            unsigned int    serial:22;  /* unique within PMD */ 
    } _pmInDom_int;

    Example 2-13. _pmInDom_int Structure in Linux

    typedef struct { 
            unsigned int    serial:22;  /* unique within PMD */         
            unsigned int    domain:8;   /* the administrative PMD */ 
            int             pad:2;
    } _pmInDom_int;

    As with metrics, the PMDA domain number is not necessarily known until run time; so the domain field must be set up when the PMDA is initialized.

    For information about how an instance domain may also be associated with more than one metric, see the pmdaInit(3) man page.

    The simple PMDA, shown in Example 2-14, has five metrics and two instance domains of three instances.

    Example 2-14. Simple PMDA

     * list of instances 
    static pmdaInstid color[] = {
        { 0, “red” }, { 1, “green” }, { 2, “blue” }
    static pmdaInstid       *timenow = NULL;
    static unsigned int     timesize = 0;
     * list of instance domains
    static pmdaIndom indomtab[] = {
    #define COLOR_INDOM     0
        { COLOR_INDOM, 3, color },
    #define NOW_INDOM       1
        { NOW_INDOM, 0, NULL },
     * all metrics supported in this PMDA - one table entry for each
    static pmdaMetric metrictab[] = {
    /* numfetch */
        { NULL,
           PMDA_PMUNITS ( 0,0,0,0,0,0) }, },
    /* color */
        { NULL,
            PMDA_PMUNITS ( 0,0,0,0,0,0) }, },
    /* time.user */
        { NULL,
            PMDA_PMUNITS ( 0, 1, 0, 0, PM_TIME_SEC, 0 ) }, },
    /* time.sys */
        { NULL,
            PMDA_PMUNITS ( 0, 1, 0, 0, PM_TIME_SEC, 0 ) }, },
    /* now */
        { NULL,
            PMDA_PMUNITS { 0,0,0,0,0,0 } }, },

    The metric simple.color is associated, via COLOR_INDOM, with the first instance domain listed in indomtab. PMDA initialization assigns the correct domain portion of the instance domain identifier in indomtab[0].it_indom and metrictab[1].m_desc.indom. This instance domain has three instances: red, green, and blue.

    The metric is associated, via NOW_INDOM, with the second instance domain listed in indomtab. PMDA initialization assigns the correct domain portion of the instance domain identifier in indomtab[1].it_indom and metrictab[4].m_desc.indom. This instance domain is dynamic and initially has no instances.

    All other metrics are singular, as specified by PM_INDOM_NULL.

    In some cases an instance domain may vary dynamically after PMDA initialization (for example,, and this requires some refinement of the default functions and data structures of the libpcp_pmda library. Briefly, this involves providing new functions that act as wrappers for pmdaInstance and pmdaFetch while understanding the dynamics of the instance domain, and then overriding the instance and fetch methods in the pmdaInterface structure during PMDA initialization.

    For the simple PMDA, the wrapper functions are simple_fetch and simple_instance, and defaults are over-ridden by the following assignments in the simple_init function:

    dp->version.two.fetch = simple_fetch;
    dp->version.two.instance = simple_instance;

    2.4. Other Issues

    Other issues include extracting the information, latency and threads of control, Name Space , PMDA help text, and management of evolution within a PMDA.

    2.4.1. Extracting the Information

    A suggested approach to writing a PMDA is to write a standalone program to extract the values from the target domain and then incorporate this program into the PMDA framework. This approach avoids concurrent debugging of two distinct problems:

    • Extraction of the data

    • Communication with PMCD

    These are some possible ways of exporting the data from the target domain:

    • Accumulate the performance data in a public shared memory segment.

    • Write the performance data to the end of a log file.

    • Periodically rewrite a file with the most recent values for the performance data.

    • Implement a protocol that allows a third party to connect to the target application, send a request, and receive new performance data.

    • For IRIX, if the data is in the operating system kernel, provide a system call (preferred) or global data (for a /dev/kmem reader) to export the performance data.

    Most of these approaches require some further data processing by the PMDA.

    2.4.2. Latency and Threads of Control

    The PCP protocols expect PMDAs to return the current values for performance metrics when requested, and with short delay (low latency). For some target domains, access to the underlying instrumentation may be costly or involve unpredictable delays (for example, if the real performance data is stored on some remote host or network device). In these cases, it may be necessary to separate probing for new performance data from servicing PMCD requests.

    An architecture that has been used successfully for several PMDAs is to create one or more sproc child processes to obtain information while the main process communicates with PMCD; see the sproc(2) man page.

    Note: The sproc(2) function is not available on Linux. You can use the exec(2)/fork(2) and setgpid(2) calls as an alternative.

    At the simplest deployment of this arrangement, the two processes may execute without synchronization. Pthreads have also been used as a more portable multithreading mechanism; see the pthreads(5) man page.

    By contrast, a complex deployment would be one in which the refreshing of the metric values must be atomic, and this may require double buffering of the data structures. It also requires coordination between parent and child processes.

    Tip: Since PMAPI is not thread-safe, only one PMDA process or thread of control should call any PMAPI functions, and this would typically be the thread servicing requests from the PMCD.

    One caveat about this style of caching PMDA--it is generally better if the PMDA converts counts to rates based upon consecutive periodic sampling from the underlying instrumentation. By exporting precomputed rate metrics with instantaneous semantics, the PMDA prevents the PCP monitor tools from computing their own rates upon consecutive PMCD fetches (which are likely to return identical values from a caching PMDA).

    2.4.3. Name Space

    The PMNS file defines the name space of the PMDA. It is a simple text file that is used during installation to expand the Name Space of the PMCD process. The format of this file is described by the pmns(4) man page.

    Client processes will not be able to access the PMDA metrics if the PMNS file is not installed as part of the PMDA installation procedure on the collector host. The installed list of metric names and their corresponding PMIDs can be found in /var/pcp/pmns/root.

    Example 2-15 shows the simple PMDA, which has five metrics:

    • Three metrics immediately under the simple node

    • Two metrics under another non-terminal node called simple.time

    Example 2-15. pmns File for the Simple PMDA

    simple {
        numfetch    SIMPLE:0:0
        color       SIMPLE:0:1
        now         SIMPLE:2:4
    simple.time {
        user        SIMPLE:1:2
        sys         SIMPLE:1:3

    Metrics that have different clusters do not have to be specified in different subtrees of the PMNS. Example 2-16 shows an alternative PMNS for the simple PMDA:

    Example 2-16. Alternate pmns File for the Simple PMDA

    simple { 
        numfetch    SIMPLE:0:0 
        color       SIMPLE:0:1 
        usertime    SIMPLE:1:2 
        systime     SIMPLE:1:3 

    In this example, the SIMPLE macro is replaced by the domain number listed in /var/pcp/pmns/stdpmid for the corresponding PMDA during installation (for the simple PMDA, this would normally be the value 253).

    2.4.4. PMDA Help Text

    For each metric defined within a PMDA, the PMDA developer is strongly encouraged to provide both terse and extended help text to describe the metric, and perhaps provide hints about the expected value ranges.

    The help text is used to describe each metric in the visualization tools and pminfo with the -T option. The help text, such as the help text for the simple PMDA in Example 2-17, is specified in a specially formatted file, normally called help. This file is converted to the expected run-time format using the newhelp command; see the newhelp(1) man page. Converted help text files are usually placed in the PMDA's directory below /var/pcp/pmdas as part of the PMDA installation procedure.

    Example 2-17. Help Text for the Simple PMDA

    The two instance domains and five metrics have a short and a verbose description. Each entry begins with a line that starts with the character “@” and is followed by either the metric name (simple.numfetch) or a symbolic reference to the instance domain number (SIMPLE.1), followed by the short description. The verbose description is on the following lines, terminated by the next line starting with “@” or end of file:

    @ SIMPLE.1 Instance domain “colour” for simple PMDA
    Universally 3 instances, “red” (0), “green” (1) and “blue” (3).
    @ SIMPLE.2 Dynamic instance domain “time” for simple PMDA
    An instance domain is computed on-the-fly for exporting current time
    information. Refer to the help text for for more details.
    @ simple.numfetch Number of pmFetch operations.
    The cumulative number of pmFetch operations directed to “simple” PMDA.
    This counter may be modified with pmstore(1).
    @ simple.color Metrics which increment with each fetch
    This metric has 3 instances, designated “red”, “green” and “blue”.
    The value of the metric is monotonic increasing in the range 0 to
    255, then back to 0.  The different instances have different starting
    values, namely 0 (red), 100 (green) and 200 (blue).
    The metric values my be altered using pmstore(1).
    @ simple.time.user Time agent has spent executing user code
    The time in seconds that the CPU has spent executing agent user code.
    @ simple.time.sys Time agent has spent executing system code
    The time in seconds that the CPU has spent executing agent system code.
    @ Time of day with a configurable instance domain
    The value reflects the current time of day through a dynamically
    reconfigurable instance domain.  On each metric value fetch request,
    the agent checks to see whether the configuration file in
    /var/pcp/pmdas/simple/simple.conf has been modified - if it has then
    the file is re-parsed and the instance domain for this metric is again
    constructed according to its contents.
    This configuration file contains a single line of comma-separated time
    tokens from this set:
      “sec”  (seconds after the minute),
      “min”  (minutes after the hour),
      “hour” (hour since midnight).
    An example configuration file could be:  sec,min,hour
    and in this case the metric would export values for the
    three instances “sec”, “min” and “hour” corresponding respectively to
    the components seconds, minutes and hours of the current time of day.
    The instance domain reflects each token present in the file, and the
    values reflect the time at which the PMDA processes the fetch.

    2.4.5. Management of Evolution within a PMDA

    Evolution of a PMDA, or more particularly the underlying instrumentation to which it provides access, over time naturally results in the appearance of new metrics and the disappearance of old metrics. This creates potential problems for PMAPI clients and PCP tools that may be required to interact with both new and former versions of the PMDA.

    The following guidelines are intended to help reduce the complexity of implementing a PMDA in the face of evolutionary change, while maintaining predictability and semantic coherence for tools using the PMAPI, and for end users of those tools.

    • Try to support as full a range of metrics as possible in every version of the PMDA. In this context, support means responding sensibly to requests, even if the underlying instrumentation is not available.

    • If a metric is not supported in a given version of the underlying instrumentation, the PMDA should respond to pmLookupDesc requests with a pmDesc structure whose type field has the special value PM_TYPE_NOSUPPORT. Values of fields other than pmid and type are immaterial, but Example 2-18 is typically benign:

      Example 2-18. Setting Values

      pmDesc dummy = { 
           PMDA_PMID(3,0),        /* pmid, fill this in */
           PM_TYPE_NOSUPPORT,     /* this is the important part */
           PM_INDOM_NULL,         /* singular,causes no problems */
          0,                     /* no semantics */
           { 0, 0, 0, 0, 0, 0 }   /* no units */

    • If a metric lacks support in a particular version of the underlying instrumentation, the PMDA should respond to pmFetch requests with a pmResult in which no values are returned for the unsupported metric. This is marginally friendlier than the other semantically acceptable option of returning an illegal PMID error or PM_ERR_PMID.

    • Help text should be updated with annotations to describe different versions of the underlying product, or product configuration options, for which a specific metric is available. This is so pmLookupText can always respond correctly.

    • The pmStore operation should fail with return status of -EACCES if a user or application tries to amend the value of an unsupported metric.

    • The value extraction, conversion, and printing functions (pmExtractValue, pmConvScale, pmAtomStr, pmTypeStr, and pmPrintValue) return the PM_ERR_CONV error or an appropriate diagnostic string, if an attempt is made to operate on a value for which type is PM_TYPE_NOSUPPORT .

      If performance tools take note of the type field in the pmDesc structure, they should not manipulate values for unsupported metrics. Even if tools ignore type in the metric's description, following these development guidelines ensures that no misleading value is ever returned; so there is no reason to call the extraction, conversion, and printing functions.

    2.5. DSO Interface

    This section describes an interface for the request handling callbacks in a PMDA. This interface is used by PMCD for communicating with DSO PMDAs, and can also be used by daemon PMDAs with pmdaMain.

    2.5.1. Overview

    Both daemon and DSO PMDAs must handle multiple request types from PMCD. A daemon PMDA communicates with PMCD using the PDU protocol, while a DSO PMDA defines callbacks for each request type. To avoid duplicating this PDU processing (in the case of a PMDA that can be installed either as a daemon or as a DSO), and to allow a consistent framework, pmdaMain can be used by a daemon PMDA as a wrapper to handle the communication protocol using the same callbacks as a DSO PMDA. This allows a PMDA to be built as both a daemon and a DSO, and then to be installed as either.

    To further simplify matters, default callbacks are declared in /usr/include/pcp/pmda.h:

    • pmdaFetch

    • pmdaProfile

    • pmdaInstance

    • pmdaDesc

    • pmdaText

    • pmdaStore

    Each callback takes a pmdaExt structure as its last argument. This structure contains all the information that is required by the default callbacks in most cases. The one exception is pmdaFetch, which needs an additional callback to instantiate the current value for each supported combination of a performance metric and an instance.

    Therefore, for most PMDAs all the communication with PMCD is automatically handled by functions in and Trivial PMDA

    The trivial PMDA uses all of the default callbacks as shown in Example 2-19. The additional callback for pmdaFetch is defined as trivial_fetchCallBack:

    Example 2-19. Request Handing Callbacks in the Trivial PMDA

    static int
    trivial_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
       __pmID_int          *idp = (__pmID_int *)&(mdesc->m_desc.pmid);
       if (idp->cluster != 0 || idp->item != 0)
           return PM_ERR_PMID;
       else if (inst != PM_IN_NULL)
           return PM_ERR_INST;
       atom->l = time(NULL);
       return 0;

    This function checks that the PMID and instance are valid, and then places the metric value for the current time into the pmAtomValue structure. The callback is set up by a call to pmdaSetFetchCallBack in trivial_init. Simple PMDA

    The simple PMDA callback for pmdaFetch is more complicated because it must support more metrics, some metrics are instantiated with each fetch, and one instance domain is dynamic. The default pmdaFetch callback, shown in Example 2-20, is replaced by simple_fetch in simple_init, which increments the number of fetches and updates the instance domain forINDOM_NOW before calling pmdaFetch:

    Example 2-20. Request Handing Callbacks in the Simple PMDA

    static int
    simple_fetch(int numpmid, pmID pmidlist[], pmResult **resp, pmdaExt *pmda)
        return pmdaFetch(numpmid, pmidlist, resp, pmda);

    The callback for pmdaFetch is defined as simple_fetchCallBack. The PMID is extracted from the pmdaMetric structure, and if valid, the appropriate field in the pmAtomValue structure is set.

    The simple.numfetch metric has no instance domain and is easily handled first as shown in Example 2-21:

    Example 2-21. simple.numfetch Metric

    static int
    simple_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
       int                 i;
       static int          oldfetch = 0;
       static struct tms   tms;
       __pmID_int          *idp = (__pmID_int *)&(mdesc->m_desc.pmid);
       if (inst != PM_IN_NULL &&
           !(idp->cluster == 0 && idp->item == 1) &&
           !(idp->cluster == 2 && idp->item == 4))
           return PM_ERR_INST;
       if (idp->cluster == 0) {
           if (idp->item == 0) {                   /* simple.numfetch */
               atom->l = numfetch;

    In Example 2-22, the inst parameter is used to specify which instance is required for the simple.color metric:

    Example 2-22. simple.color Metric

           else if (idp->item == 1) {              /* simple.color */
                switch (inst) {
                case 0:                             /* red */
                    red = (red + 1) % 256;
                    atom->l = red;
                case 1:                             /* green */
                    green = (green + 1) % 256;
                    atom->l = green;
                case 2:                             /* blue */
                    blue = (blue + 1) % 256;
                    atom->l = blue;
                    return PM_ERR_INST;
               return PM_ERR_PMID;

    In Example 2-23, the simple.time metric is in a second cluster and has a simple optimization to reduce the overhead of calling times twice on the same fetch and return consistent values from a single call to times when both metrics simple.time.user and simple.time.sys are requested in a single pmFetch. The previous fetch count is used to determine if the tms structure should be updated:

    Example 2-23. simple.time Metric

       else if (idp->cluster == 1) {               /* simple.time */
           if (oldfetch < numfetch) {
               oldfetch = numfetch;
           if (idp->item == 2)                     /* simple.time.user */
               atom->d = (tms.tms_utime / (double)CLK_TCK);
           else if (idp->item == 3)                /* simple.time.sys */
               atom->d = (tms.tms_stime / (double)CLK_TCK);
               return PM_ERR_PMID;

    In Example 2-24, the metric is in a third cluster and uses inst again to select a specific instance from the INDOM_NOW instance domain:

    Example 2-24. Metric

        else if (idp->cluster == 2) {
            if (idp->item == 4) {                 /* */
                /* this loop will always match one of the named */
                /* time constants from the timeslices structure */
                for (i = 0; i < num_timeslices; i++) {
                    if (inst == timeslices[i].inst_id) {
                        atom->l = timeslices[i].tm_field;
                if (i == num_timeslices)
                    return PM_ERR_INST;
                return PM_ERR_PMID;
        } simple_store in the Simple PMDA

    The simple PMDA permits some of the metrics it supports to be modified by pmStore as shown in Example 2-25. For additional information, see the pmstore(1) man page.

    Example 2-25. simple_store in the Simple PMDA

    The pmdaStore callback (which returns -EACCESS to indicate no metrics can be altered) is replaced by simple_store in simple_init. This replacement function must take the same arguments so that it can be assigned to the function pointer in the pmdaInterface structure.

    The function traverses the pmResult and checks the cluster and unit of each PMID to ensure that it corresponds to a metric that can be changed. Checks are made on the values to ensure they are within range before being assigned to variables in the PMDA that hold the current values for exported metrics:

    static int
    simple_store(pmResult *result, pmdaExt *pmda)
       int         i, j, val, sts = 0;
       pmAtomValue av;
       pmValueSet  *vsp = NULL;
       __pmID_int  *pmidp = NULL;
       for (i = 0; i < result->numpmid; i++) {
           vsp = result->vset[i];
           pmidp = (__pmID_int *)&vsp->pmid;
           if (pmidp->cluster == 0) {  /* storable metrics are cluster0 */
               switch (pmidp->item) {
                   case 0:                           /* simple.numfetch */
                       val = vsp->vlist[0].value.lval;
                       if (val < 0) {
                           sts = PM_ERR_SIGN;
                           val = 0;
                       numfetch = val;
                   case 1:                             /* simple.color */
                       for (j = 0; j < vsp->numval && sts == 0; j++) {
                           val = vsp->vlist[j].value.lval;
                           if (val < 0) {
                               sts = PM_ERR_SIGN;
                               val = 0;
                           } if (val > 255) {
                               sts = PM_ERR_CONV;
                               val = 255;

    The simple.color metric has an instance domain that must be searched because any or all instances may be specified. Any instances that are not supported in this instance domain should cause an error value of PM_ERR_INST to be returned as shown in Example 2-26:

    Example 2-26. simple.color and PM_ERR_INST Errors

                           switch (vsp->vlist[j].inst) {
                               case 0:                         /* red */
                                   red = val;
                               case 1:                         /* green */
                                   green = val;
                               case 2:                         /* blue */
                                   blue = val;
                                   sts = PM_ERR_INST;

    Any other PMIDs in cluster 0 that are not supported by the simple PMDA should result in an error value of PM_ERR_PMID as shown in Example 2-27:

    Example 2-27. PM_ERR_PMID Errors

                       sts = PM_ERR_PMID;

    Any metrics that cannot be altered should generate an error value of -EACCES, and metrics not supported by the PMDA should result in an error value of PM_ERR_PMID as shown in Example 2-28:

    Example 2-28. -EACCES and PM_ERR_PMID Errors

           else if ((pmidp->cluster == 1 &&
                    (pmidp->item == 2 || pmidp->item == 3)) ||
                    (pmidp->cluster == 2 && pmidp->item == 4)) {
               sts = -EACCES;
           else {
               sts = PM_ERR_PMID;
       return sts;

    The structure pmdaExt argument is not used by the simple_store function above. Return Codes for pmdaFetch Callbacks

    In PMDA_INTERFACE_1 and PMDA_INTERFACE_2, the return codes for the pmdaFetch callback function are defined:



    < 0 

    Error code (for example, PM_ERR_PMID, PM_ERR_INST or PM_ERR_AGAIN)



    In PMDA_INTERFACE_3, the return codes for the pmdaFetch callback function are defined:



    < 0 

    Error code (for example, PM_ERR_PMID, PM_ERR_INST)


    Metric value not currently available

    > 0 


    2.5.2. PMDA Structures

    PMDA structures used with the pcp_pmda library are defined in /usr/include/pcp/pmda.h. Example 2-29 and Example 2-30 describe the pmdaInterface and pmdaExt structures.

    Example 2-29. pmdaInterface Structure

    The callbacks must be specified in a pmdaInterface structure:

    typedef struct {
       int domain;     /* set/return performance metrics domain id here */
       struct {
           unsigned int pmda_interface : 8;  /* PMDA DSO version */
           unsigned int pmapi_version : 8;   /* PMAPI version */
           unsigned int flags : 16;          /* usage TBD */
       } comm;             /* set/return communication and version info */
       int status;         /* return initialization status here */
       union {
    * Interface Version 2 (PCP 2.0) or later 
           struct {
               pmdaExt *ext;
               int     (*profile)(__pmProfile *, pmdaExt *);
               int     (*fetch)(int, pmID *, pmResult **, pmdaExt *);
               int     (*desc)(pmID, pmDesc *, pmdaExt *);
               int     (*instance)(pmInDom, int, char *, __pmInResult **,
                       pmdaExt *);
               int     (*text)(int, int, char **, pmdaExt *);
               int     (*store)(pmResult *, pmdaExt *);
           } two;
       } version;
    } pmdainterface;

    This structure is passed by PMCD to a DSO PMDA as an argument to the initialization function. This structure supports two versions--the second version adds support for the pmdaExt structure. Protocol version one is for backwards compatibility only, and should not be used in any new PMDA.

    Example 2-30. pmdaExt Stucture

    Additional PMDA information must be specified in a pmdaExt structure:

    typedef struct {
       unsigned int e_flags;       /* used internally within libpcp_pmda */
       void        *e_ext;         /* used internally within libpcp_pmda */
       char        *e_sockname;    /* socket name to pmcd */
       char        *e_name;        /* name of this pmda */
       char        *e_logfile;     /* path to log file */
       char        *e_helptext;    /* path to help text */
       int         e_status;       /* =0 is OK */
       int         e_infd;         /* input file descriptor from pmcd */
       int         e_outfd;        /* output file descriptor to pmcd */
       int         e_port;         /* port to pmcd */
       int         e_singular;     /* =0 for singular values */
       int         e_ordinal;      /* >=0 for non-singular values */
       int         e_direct;       /* =1 if pmid map to meta table */
       int         e_domain;       /* metrics domain */
       int         e_nmetrics;     /* number of metrics */
       int         e_nindoms;      /* number of instance domains */
       int         e_help;         /* help text comes via this handle */
       __pmProfile *e_prof;        /* last received profile */
       pmdaIoType  e_io;           /* connection type to pmcd */
       pmdaIndom   *e_indoms;      /* instance domain table */
       pmdaIndom   *e_idp;         /* instance domain expansion */
       pmdaMetric  *e_metrics;     /* metric description table */
       pmdaResultCallBack e_resultCallBack; /* to clean up pmResult after fetch */
       pmdaFetchCallBack  e_fetchCallBack;  /* to assign metric values in fetch */
       pmdaCheckCallBack  e_checkCallBack;  /* callback on receipt of a PDU */
       pmdaDoneCallBack   e_doneCallBack;   /* callback after PDU is processed */
    } pmdaExt;

    The pmdaExt structure contains filenames, pointers to tables, and some variables shared by several functions in the pcp_pmda library. All fields of the pmdaInterface and pmdaExt structures can be correctly set by PMDA initialization functions; see the pmdaDaemon(3), pmdaDSO(3), pmdaGetOpt(3), pmdaInit(3), and pmdaConnect(3) man pages for a full description of how various fields in these structures may be set or used by pcp_pmda library functions.

    2.6. Initializing a PMDA

    Several functions are provided to simplify the initialization of a PMDA. These functions, if used, must be called in a strict order so that the PMDA can operate correctly.

    2.6.1. Overview

    The initialization process for a PMDA involves opening help text files, assigning callback function pointers, adjusting the metric and instance identifiers to the correct domains, and much more. The initialization of a daemon PMDA also differs significantly from a DSO PMDA, since the pmdaInterface structure is initialized by main or the PMCD process, respectively.

    2.6.2. Common Initialization

    As described in Section 2.2.2, an initialization function is provided by a DSO PMDA and called by PMCD. Using the standard PMDA wrappers, the same function can also be used as part of the daemon PMDA initialization. This PMDA initialization function performs the following tasks:

    • Assigning callback functions to the function pointer interface of pmdaInterface

    • Assigning pointers to the metric and instance tables from pmdaExt

    • Opening the help text files

    • Assigning the domain number to the instance domains

    • Correlating metrics with their instance domains

    If the PMDA uses the common data structures defined for the pcp_pmda library, most of these requirements can be handled by the default pmdaInit function; see the pmdaInit(3) man page.

    Because the initialization function is the only initialization opportunity for a DSO PMDA, the common initialization function should also perform any DSO-specific functions that are required. A default implementation of this functionality is provided by the pmdaDSO function; see the pmdaDSO(3) man page. Trivial PMDA

    Example 2-31 shows the trivial PMDA, which has no instances (that is, all metrics have singular values) and a single callback. This callback is for the pmdaFetch function called trivial_fetchCallBack; see the pmdaFetch(3) man page:

    Example 2-31. Initialization in the Trivial PMDA

    void trivial_init(pmdaInterface *dp)
       pmdaSetFetchCallBack(dp, trivial_fetchCallBack);
       pmdaInit(dp, NULL, 0,
                metrictab, sizeof(metrictab)/sizeof(metrictab[0]));

    The trivial PMDA is always installed as a daemon PMDA. Simple PMDA

    In Example 2-32, the simple PMDA uses its own callbacks to handle PDU_FETCH and PDU_RESULT request PDUs (for pmFetch and pmStore operations respectively), as well as providing pmdaFetch with the callback simple_fetchCallBack.

    Example 2-32. Initialization in the Simple PMDA

    static int      isDSO = 1;              /* =0 I am a daemon */
    void simple_init(pmdaInterface *dp)
       if (isDSO)
           pmdaDSO(dp, PMDA_INTERFACE_2, “simple DSO”,
       if (dp->status != 0)
       dp->version.two.fetch = simple_fetch;
       dp-> = simple_store;
       dp->version.two.instance = simple_instance;
       pmdaSetFetchCallBack(dp, simple_fetchCallBack);
       pmdaInit(dp, indomtab, sizeof(indomtab)/sizeof(indomtab[0]),
                metrictab, sizeof(metrictab)/sizeof(metrictab[0]));

    The simple PMDA may be installed either as a daemon PMDA or a DSO PMDA. The static variable isDSO indicates whether the PMDA is running as a DSO or as a daemon. A daemon PMDA should change the value of this variable to 0 in main.

    2.6.3. Daemon Initialization

    In addition to the initialization function that can be shared by a DSO and a daemon PMDA, a daemon PMDA must also meet the following requirements:

    • Create the pmdaInterface structure that is passed to the initialization function

    • Parse any command-line arguments

    • Open a log file (a DSO PMDA uses PMCD's log file)

    • Set up the IPC connection between the PMDA and the PMCD process

    • Handle incoming PDUs

    All these requirements can be handled by default initialization functions in the pcp_pmda library; see the pmdaDaemon(3), pmdaGetOpt(3), pmdaOpenLog(3), pmdaConnect(3), and pmdaMain(3) man pages.

    The simple PMDA requires no additional command-line arguments other than those handled by pmdaGetOpt as shown in Example 2-33. For additional information, see the pmdaGetOpt(3) man page.

    Example 2-33. main in the Simple PMDA

    main(int argc, char **argv)
       int                 err = 0;
       pmdaInterface       dispatch;
       char                *p;
       /* trim cmd name of leading directory components */
       pmProgname = argv[0];
       for (p = pmProgname; *p; p++) {
           if (*p == `/')
               pmProgname = p+1;
       isDSO = 0;
       pmdaDaemon(&dispatch, PMDA_INTERFACE_2, pmProgname, SIMPLE,
                  “simple.log”, “/var/pcp/pmdas/simple/help”);
       if (pmdaGetOpt(argc, argv, “D:d:i:l:pu:?”, &dispatch, &err)) != EOF)
       if (err)

    2.7. Testing and Debugging a PMDA

    Ensuring the correct operation of a PMDA can be difficult, because the responsibility of providing metrics to the requesting PMCD process and simultaneously retrieving values from the target domain requires nearly real-time communication with two modules beyond the PMDA's control. Some tools are available to assist in this important task.

    2.7.1. Overview

    Thoroughly testing a PMDA with PMCD is difficult, although testing a daemon PMDA is marginally simpler than testing a DSO PMDA. If a DSO PMDA exits, PMCD also exits because they share a single address space and control thread. If the PMDA dumps core, dbx and related tools on IRIX or gdb(1) on Linux cannot reasonably explore the generated core image, which includes the PMCD image and any other active DSO PMDAs. For more information, see the dbx(1) or gdb(1)man page.

    The difficulty in using PMCD to test a daemon PMDA results from PMCD requiring timely replies from the PMDA in response to request PDUs. Although a timeout period can be set in /etc/config/pmcd.options on IRIX or /var/pcp/config/pmcd/pmcd.options on Linux, attaching dbx or gdb to the PMDA process (or any other long delay) might cause an already running PMCD to close its connection with the PMDA. If timeouts are disabled, PMCD could wait forever to connect with the PMDA.

    If you suspect a PMDA has been terminated due to a timeout failure, check the PMCD log file, usually /var/adm/pcplog/pmcd.log for IRIX or /var/log/pcp/pmcd/pmcd.log for Linux.

    A more robust way of testing a PMDA is to use the dbpmda tool, which is similar to PMCD except that dbpmda provides complete control over the PDUs that are sent to the PMDA, and there are no time limits--it is essentially an interactive debugger for exercising a PMDA. See the dbpmda(3) man page for details.

    In addition, careful use of PCP debugging flags can produce useful information concerning a PMDA's behavior; see the pmapi(3) and pmdbg(1) man pages for a discussion of the PCP debugging and tracing framework.

    2.7.2. Debugging Information

    You can activate debugging flags in PMCD and most other PCP tools with the -D command-line option. Supported flags can be listed with the pmdbg command; see the pmdbg(1) man page. Setting the debug flag for PMCD in /etc/config/pmcd.options on IRIX or /var/pcp/config/pmcd/pmcd.options on Linux might generate too much information to be useful, especially if there are other clients and PMDAs connected to the PMCD process.

    The PMCD debugging flag can also be changed dynamically by storing a new value into the metric pmcd.control.debug:

    # pmstore pmcd.control.debug 5

    Most of the pcp_pmda library functions log additional information if the DBG_TRACE_LIBPMDA flag is set within the PMDA; see the pmda(3) man page. The command-line argument -D is trapped by pmdaGetOpt to set the global debugging control variable pmDebug. Adding tests within the PMDA for the DBG_TRACE_APPL0, DBG_TRACE_APPL1, and DBG_TRACE_APPL2 trace flags permits different levels of information to be logged to the PMDA's log file.

    All diagnostic, debugging, and tracing output from a PMDA should be written to the standard error stream. By convention, all debugging information is enclosed by preprocessor #ifdef DEBUG statements so that they can be compiled out of the program at a later stage, if required.

    Adding this segment of code to the simple_store metric causes a log message to be sent to the current log file whenever pmstore attempts to change simple.numfetch and pmDebug has the DBG_TRACE_APPL0 flag set as shown in Example 2-34:

    Example 2-34. simple.numfetch in the Simple PMDA

       case 0: /* simple.numfetch */ 
           val = vsp->vlist[0].value.lval; 
           if (val < 0) { 
               sts = PM_ERR_SIGN; 
               val = 0; 
    #ifdef DEBUG 
           if (pmDebug & DBG_TRACE_APPL0) { 
                    "simple: %d stored into numfetch", val); 
           numfetch = val; 

    For a description of pmstore, see the pmstore(1) man page.

    2.7.3. dbpmda Debug Utility

    The dbpmda utility provides a simple interface to the PDU communication protocol. It allows daemon and DSO PMDAs to be tested with most request types, while the PMDA process may be monitored with dbx, par on IRIX or gdb, strace on Linux, and other diagnostic tools. The dbpmda(1) man page contains a sample session with the simple PMDA.

    2.8. Integration of PMDA

    Several steps are required to install (or remove) a PMDA from a production PMCD environment without affecting the operation of other PMDAs or related visualization and logging tools.

    The PMDA typically would have its own directory below /var/pcp/pmdas into which several files would be installed. In the description in Section 2.8.1, the PMDA of interest is assumed to be known by the name newbie, hence the PMDA directory would be /var/pcp/pmdas/newbie.

    Note: Any installation or removal of a PMDA involves updating files and directories that are typically well protected. Hence the procedures described in this section must be executed as the superuser.

    2.8.1. Installing a PMDA

    A PMDA is fully installed when these tasks are completed:

    • Help text has been installed in a place where the PMDA can find it, usually in the PMDA directory /var/pcp/pmdas/newbie.

    • The name space has been updated in the /var/pcp/pmns directory.

    • The PMDA binary has been installed, usually in the directory /var/pcp/lib for a DSO PMDA, or in the PMDA directory /var/pcp/pmdas/newbie for a daemon PMDA.

    • The /etc/pmcd.conf file on IRIX or the /var/pcp/config/pmcd/pmcd.conf file on Linux has been updated.

    • The PMCD process has been restarted or notified (with a SIGHUP signal) that the new PMDA exists.

    The Makefile should include an install target to compile and link the PMDA (as a DSO, or a daemon or both) in the PMDA directory, and in the case of a DSO PMDA, install the shared library in /var/pcp/lib for IRIX and /usr/share/pcp/lib for Linux. The clobber target should remove any files created as a by-product of the install target.

    You may wish to use /var/pcp/pmdas/simple/Makefile as a template for constructing a new PMDA Makefile; changing the assignment of IAM from simple to newbie would account for most of the required changes.

    Since the object format of a DSO PMDA must match the object format of PMCD, which in turn must match the object format of the booted operating system kernel, there might be multiple DSO targets in the Makefile. For an example on IRIX, see targets mips_o32.pmda_$(IAM).so, mips_n32.pmda_$(IAM).so, and mips_64.pmda_$(IAM).so for the simple PMDA.

    The Install script should make use of the generic procedures defined in the script /usr/pcp/lib/ on IRIX and /usr/share/pcp/lib/ on Linux, and may be as straightforward as the one used for the trivial PMDA, shown in Example 2-35:

    Example 2-35. Install Script for the Trivial PMDA

    # Get the common procedures and variable assignments
    . /usr/pcp/lib/
    # The name of the PMDA
    # Do it
    exit 0

    The variables, shown in Table 2-1, may be assigned values to modify the behavior of the pmdaSetup and pmdainstall procedures from /usr/pcp/lib/ or /usr/share/pcp/lib/ on Linux.

    Table 2-1. Variables to Control Behavior of Generic Procedures

    Shell Variable




    Name of the PMDA; assignment to this variable is mandatory.

    Example: iam=newbie



    Can this PMDA be installed as a DSO?



    Can this PMDA be installed as a daemon?



    If installed as a daemon PMDA, is the default IPC via pipes?



    If installed as a daemon PMDA, is the default IPC via an Internet socket?



    If installed as a daemon PMDA, and the IPC method uses an Internet socket, the default port number.



    IPC style for PDU exchanges involving a daemon PMDA; binary or text.



    Delay in seconds between installing PMDA and checking if metrics are available.



    Additional command-line arguments passed to a daemon PMDA.



    Version of the libpcp_pmda library required, used to determine the version for generating help text files.



    The name of the PMNS file (by default relative to the PMDA directory).



    First-level name for this PMDA's metrics in the PMNS.



    The name of the help file (by default relative to the PMDA directory).



    The name of the executable for a daemon PMDA.



    The name of the shared library for a DSO PMDA.



    The name of the initialization function for a DSO PMDA.



    The numerical PMDA domain number (from domain.h).



    The symbolic name of the PMDA domain number (from domain.h).


    In addition, the variables do_pmda and do_check will be set to reflect the intention to install the PMDA (as opposed to install just the PMNS) and to check the availability of the metrics once the PMDA is installed. By default, each variable is true; however, the command-line options -N and -Q to Install may be used to set the variables to false, as follows: do_pmda (-N) and do_check (-N or -Q).

    The variables may also have their assignments changed by the user's response to the common prompt as shown in Example 2-36:

    Example 2-36. Changing Variable Assignments

    You will need to choose an appropriate configuration for installation 
    of the ... Performance Metrics Domain Agent (PMDA).
      collector   collect performance statistics on this system
      monitor     allow this system to monitor local and/or remote systems
      both        collector and monitor configuration for this system

    Obviously, for anything but the most trivial PMDA, after calling the pmdaSetup procedure, the Install script should also prompt for any PMDA-specific parameters, which are typically accumulated in the args variable and used by the pmdainstall procedure.

    The detailed operation of the pmdainstall procedure involves the following tasks:

    • Using default assignments, and interaction where ambiguity exists, determine the PMDA type (DSO or daemon) and the IPC parameters, if any.

    • Copy the $pmns_source file, replacing symbolic references to SYMDOM by the desired numeric domain number from domain.

    • Merge the PMDA's name space into the PCP name space at the non-leaf node identified by $pmns_name.

    • If any pmchart views can be found (files with names ending in “.pmchart”), copy these to the standard directory (/var/pcp/config/pmchart) with the “.pmchart” suffix removed.

    • Create new help files from $help_source after replacing symbolic references to SYMDOM by the desired numeric domain number from domain.

    • Terminate the old daemon PMDA, if any.

    • Use the Makefile to build the appropriate executables.

    • Add the PMDA specification to PMCD's configuration file (/etc/pmcd.conf).

    • Notify PMCD. To minimize the impact on the services PMCD provides, sending a SIGHUP to PMCD forces it to reread the configuration file and start, restart, or remove any PMDAs that have changed since the file was last read.

    • Check that the metrics from the new PMDA are available.

    There are some PMDA changes that may trick PMCD into thinking nothing has changed, and not restarting the PMDA. Most notable are changes to the PMDA executable. In these cases, you may need to explicitly remove the PMDA as described in Section 2.8.3, or more drastically, restart PMCD on IRIX as follows:

    # /etc/init.d/pcp start

    Restart PMCD on Linux as follows:

    # /etc/rc.d/init.d/pcp start

    The files /var/pcp/pmdas/*/Install provide a wealth of examples that may be used to construct a new PMDAInstall script.

    2.8.2. Upgrading a PMNS to Include Metrics from a New PMDA

    When invoked with a -N command-line option, the PMDA Install script may be used to update the PMNS without installing the PMDA. This is typically used on a monitoring system to populate the local PMNS with the names of the performance metrics from a PMDA installed on a remote host. The -N option also installs pmchart views useful on a monitoring system.

    2.8.3. Removing a PMDA

    The simplest way to stop a PMDA from running, apart from killing the process, is to remove the entry from /etc/pmcd.conf and signal PMCD (with SIGHUP) to reread its configuration file. To completely remove a PMDA requires the reverse process of the installation, including an update of the Performance Metrics Name Space (PMNS).

    This typically involves a Remove script in the PMDA directory that uses the same common procedures as the Install script described Section 2.8.1.

    The /var/pcp/pmdas/*/Remove files provide a wealth of examples that may be used to construct a new PMDA Remove script.

    2.8.4. Configuring PCP Tools

    Most PCP tools have their own configuration file format for specifying which metrics to view or to log. By using canned configuration files that monitor key metrics of the new PMDA, users can quickly see the performance of the target system, as characterized by key metrics in the new PMDA.

    Any configuration files that are created should be kept with the PMDA and installed into the appropriate directories when the PMDA is installed.

    As with all PCP customization, some of the most valuable tools can be created by defining views, scenes, and control-panel layouts that combine related performance metrics from multiple PMDAs or multiple hosts.

    Parameterized alarm configurations can be created using the pmieconf facilities; see the pmieconf(1) and pmie(1) man pages. In addition, pmie rules involving metrics from the new PMDA may be created directly.

    Daily logs can be specified in pmlogger configuration files, or with the pmlogger_daily mechanism; see the pmlogger(1) and pmlogger_daily(1) man pages. The services of pmsnap may be used to incorporate the new performance metrics into charts that may be periodically regenerated and published via a World Wide Web server.