All PMDAs are launched and controlled by the PMCD process on the local host. PMCD receives requests from the monitoring tools and forwards them to the PMDAs. Responses, when required, are returned through PMCD to the clients. The requests fall into a small number of categories, and the PMDA must handle each request type. For a DSO PMDA, each request type corresponds to a method in the agent. For a daemon PMDA, each request translates to a message or protocol data unit (PDU) that may be sent to a PMDA from PMCD.

For daemon PMDA, the following request PDUs must be supported:

Each PMDA is associated with a unique domain number that is encoded in the domain field of metric and instance identifiers, and PMCD uses the domain number to determine which PMDA can handle the components of any given client request.

Each PMDA is required to implement a function that handles each of the request types. By implementing these functions as library functions, a PMDA can be implemented as a dynamically shared object (DSO) and attached by PMCD at run time with the dlopen call; see the dlopen(3) man page. This eliminates the need for an IPC layer (typically a UNIX pipe) between each PMDA and PMCD, because each request becomes a function call rather than a message exchange. The required library functions are detailed in Section 2.5.

A PMDA that interacts with PMCD in this fashion must abide by a formal initialization protocol so that PMCD can discover the location of the library functions that are subsequently called with function pointers. When a DSO PMDA is installed, the PMCD configuration file, /etc/pmcd.conf on IRIX; /var/pcp/config/pmcd/pmcd.conf, is updated to reflect the domain and name of the PMDA, the location of the shared object, and the name of the initialization function. The initialization sequence is discussed in Section 2.6.

As superuser, install the simple PMDA as a DSO, as shown in Example 2-1 and Example 2-2, and observe the changes in the PMCD configuration file. The output may differ slightly depending on the other PMDAs you have installed.

# cd /var/pcp/pmdas/simple
# cat /etc/pmcd.conf
# Name  Id      IPC     IPC Params      File/Cmd 
irix    1       dso     irix_init       libirixpmda.so 
pmcd    2       dso     pmcd_init       pmda_pmcd.so 
proc    3       dso     proc_init       pmda_proc.so 
# ./Install
You will need to choose an appropriate configuration for installation 
of the “simple” Performance Metrics Domain Agent (PMDA).
collector   collect performance statistics on this system
  monitor     allow this system to monitor local and/or remote systems
  both         collector and monitor configuration for this system
Please enter c(ollector) or m(onitor) or b(oth) [b] both

Updating the Performance Metrics Name Space (PMNS) ... 
Installing pmchart view(s) ...
Install simple as a daemon or dso agent? [daemon] dso
...
Check simple metrics have appeared ... 5 metrics and 9 values
# cat /etc/pmcd.conf
# Name  Id      IPC     IPC Params      File/Cmd 
irix    1       dso     irix_init       libirixpmda.so 
pmcd    2       dso     pmcd_init       pmda_pmcd.so 
proc    3       dso     proc_init       pmda_proc.so 
simple  253     dso     simple_init     pmda_simple.so

cat /var/pcp/config/pmcd/pmcd.conf
# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
pmcd    2       dso     pmcd_init       /var/pcp/pmdas/pmcd/pmda_pmcd.so
linux   60      dso     linux_init      /var/pcp/pmdas/linux/pmda_linux.so

snia 63 pipe binary /var/pcp/pmdas/snia/pmdasnia -d 63
simple  254     dso     simple_init     pmda_simple.so

As can be seen from the contents of /var/pcp/config/pmcd/pmcd.conf, the DSO version of the simple PMDA is in a library named pmda_simple.so and has an initialization function called simple_init. The domain of the simple PMDA is 254, as shown in the column headed Id.

A DSO PMDA provides the most efficient communication between the PMDA and PMCD. This approach has some disadvantages resulting from the DSO PMDA being the same process as PMCD:

Consequently, many PMDAs are implemented as a daemon process.

The libpcp_pmda library is designed to allow simple implementation of a PMDA that runs as a separate process. The library functions provide a message passing layer acting as a generic wrapper that accepts PDUs, makes library calls using the standard DSO PMDA interface, and sends PDUs. Therefore, you can implement a PMDA as a DSO and then install it as either a daemon or a DSO, depending on the presence or absence of the generic wrapper.

The PMCD process launches a daemon PMDA with fork and execv. You can easily connect a pipe to the PMDA using standard input and output. The PMCD process may also connect to a daemon PMDA using TCP/IP or UNIX domain sockets; see the inet(7) or unix(7) man page.

As superuser, install the simple PMDA as a daemon process as shown in Example 2-3 for IRIX and Example 2-4 for Linux. As in Example 2-1, the output may differ due to other PMDAs already installed.

# cd /var/pcp/pmdas/simple
# ./Install 
... 
Install simple as a daemon or dso agent? [daemon] daemon 
PMCD should communicate with the daemon via pipe or socket? [pipe] pipe
...
# cat /etc/pmcd.conf 
# Name  Id   IPC     IPC Params File/Cmd 
irix    1    dso     irix_init  libirixpmda.so 
pmcd    2    dso     pmcd_init  pmda_pmcd.so 
proc    3    dso     proc_init  pmda_proc.so 
simple  253  pipe    binary     /var/pcp/pmdas/simple/pmdasimple -d 253

# cd /var/pcp/pmdas/simple
# ./Install
... 
Install simple as a daemon or dso agent? [daemon] daemon 
PMCD should communicate with the daemon via pipe or socket? [pipe] pipe
...
# cat /var/pcp/config/pmcd/pmcd.conf 
# Performance Metrics Domain Specifications
# 
# This file is automatically generated during the build
# Name  Id      IPC     IPC Params      File/Cmd
pmcd    2       dso     pmcd_init       /var/pcp/pmdas/pmcd/pmda_pmcd.so
linux   60      dso     linux_init      /var/pcp/pmdas/linux/pmda_linux.so
snia 63 pipe binary /var/pcp/pmdas/snia/pmdasnia -d 63 
simple  253  pipe    binary     /var/pcp/pmdas/simple/pmdasimple -d 253

When either the cost or latency associated with collecting performance metrics is high, the PMDA implementer may choose to trade off the currency of the performance data to reduce the PMDA resource demands or the fetch latency time.

One scheme for doing this is called a caching PMDA, which periodically instantiates values for the performance metrics and responds to each request from PMCD with the most recently instantiated (or cached) values, as opposed to instantiating current values on demand when the PMCD asks for them.

The Cisco PMDA is an example of a caching PMDA. For additional information, see the contents of the /var/pcp/pmdas/cisco directory and the pmdacisco(1) man page.

Domains are autonomous performance areas, such as the operating system or a layered service or a particular application. Metrics are raw performance data for a domain, and typically quantify activity levels, resource utilization or quality of service. Instances are sets of related metrics, as for multiple processors, or multiple service classes, or multiple transaction types.

PCP employs the following simple and uniform data model to accommodate the demands of performance metrics drawn from multiple domains:

The selection of metrics and instances is an important design decision for a PMDA implementer. The metrics and instances for a target domain should have the following qualities:

For each metric, you should also consider these questions:

As with all programming tasks, expect to refine the choice of metrics and instances several times during the development of the PMDA.

Each PMDA must be uniquely identified by PMCD so that requests from clients can be efficiently routed to the appropriate PMDA. The unique identifier, the PMDA's domain, is encoded within the metrics and instance domain identifiers so that they are associated with the correct PMDA, and so that they are unique, regardless of the number of PMDAs that are connected to the PMCD process.

The default domain number for each PMDA is defined in /var/pcp/pmns/stdpmid. This file is a simple table of PMDA names and their corresponding domain number. However, a PMDA does not have to use this domain number--the file is only a guide to help avoid domain number clashes when PMDAs are installed and activated.

The domain number a PMDA uses is passed to the PMDA by PMCD when the PMDA is launched. Therefore, any data structures that require the PMDA's domain number must be set up when the PMDA is initialized, rather than declared statically. The protocol for PMDA initialization provides a standard way for a PMDA to implement this run-time initialization.

---

Tip: Although uniqueness of the domain number in the /etc/pmcd.conf control file used by PMCD is all that is required for successful starting of PMCD and the associated PMDAs, the developer of a new PMDA is encouraged to add the default domain number for each new PMDA to the /var/pcp/pmns/stdpmid.local file and then to run the Make.stdpmid script in /var/pcp/pmns to recreate /var/pcp/pmns/stdpmid; this file acts as a repository for documenting the known default domain numbers.

---

A PMDA provides support for a collection of metrics. In addition to the obvious performance metrics, and the measures of time, activity and resource utilization, the metrics should also describe how the target domain has been configured, as this can greatly affect the correct interpretation of the observed performance. For example, metrics that describe network transfer rates should also describe the number and type of network interfaces connected to the host.

In addition, the metrics should describe how the PMDA has been configured. For example, if the PMDA was periodically probing a system to measure quality of service, there should be metrics for the delay between probes, the number of probes attempted, plus probe success and failure counters. It may also be appropriate to allow values to be stored (see the pmstore(1) man page) into the delay metric, so that the delay used by the PMDA can be altered dynamically.

2.3.3.1. Data Structures

Each metric must be described in a pmDesc structure; see the pmLookupDesc(3) man page:

typedef struct { pmID pmid; /* unique identifier */ int type; /* base data type */ pmInDom indom; /* instance domain */ int sem; /* semantics of value */ pmUnits units; /* dimension and units */ } pmDesc;

This structure contains the following fields:

pmid	A unique identifier, Performance Metric Identifier (PMID), that differentiates this metric from other metrics across the union of all PMDAs
type	A data type indicator showing whether the format is an integer (32 or 64 bit, signed or unsigned); float; double; string; or arbitrary aggregate of binary data
indom	An instance domain identifier that links this metric to an instance domain
sem	An encoding of the value's semantics (counter, instantaneous, or discrete)
units	A description of the value's units based on dimension and scale in the three orthogonal dimensions of space, time, and count (or events)

Symbolic constants of the form PM_TYPE_*, PM_SEM_*, PM_SPACE_*, PM_TIME_*, and PM_COUNT_* are defined in the /usr/include/pcp/pmapi.h file. You may use them to initialize the elements of a pmDesc structure. The pmID type is an unsigned integer that can be safely cast to a _pmID_int structure, which contains fields defining the metric's (PMDA's) domain, cluster, and item number as shown in Example 2-5 for IRIX and Example 2-6 for Linux:

Example 2-5. _pmID_int Structure in IRIX

typedef struct { int pad:2; unsigned int domain:8; unsigned int cluster:12; unsigned int item:10; } _pmID_int;

Example 2-6. _pmID_int Structure in Linux

typedef struct { unsigned int item:10; unsigned int cluster:12; unsigned int domain:8; int pad:2; } _pmID_int;

For additional information, see the /usr/include/pcp/impl.h file.

The pad field should be ignored. The domain number should be set at run time when the PMDA is initialized. The PMDA_PMID macro defined in /usr/include/pcp/pmapi.h can be used to set the cluster and item fields at compile time, as these should always be known and fixed for a particular metric.

---

Note: The three components of the PMID should correspond exactly to the three-part definition of the PMID for the corresponding metric in the PMNS described in Section 2.4.3.

---

A table of pmdaMetric structures should be defined within the PMDA, with one structure per metric as shown in Example 2-7.

Example 2-7. pmdaMetric Structure

typedef struct { void *m_user; /* for users external use */ pmDesc m_desc; /* metric description */ } pmdaMetric;

This structure contains a pmDesc structure and a handle that allows PMDA-specific structures to be associated with each metric. For example, m_user could be a pointer to a global variable containing the metric value, or a pointer to a function that may be called to instantiate the metric's value.

The trivial PMDA, shown in Example 2-8, has only a singular metric (that is, no instance domains):

Example 2-8. Trivial PMDA

static pmdaMetric metrictab[] = { /* time */ { (void *)0, { PMDA_PMID(0,1), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT, PMDA_PMUNITS (0, 1, 0, 0, PM_TIME_SEC, 0) ), ) };

This single metric (trivial.time) has the following:

• A PMID with a cluster of 0 and an item of 1

• An unsigned 32-bit integer (PM_TYPE_U32 )

• A singular value and hence no instance domain (PM_INDOM_NULL)

• An instantaneous semantic value (PM_SEM_INSTANT)

• Dimension “time” and the units “seconds”

Timestamps:         1   3   5   7   9  11 
Value:             10  30  60  80  90 N/A

Timestamps:         1   3   5   7   9  11 
Semantics: 
Counter           N/A  10  15  10   5 N/A 
Instantaneous      10  30  60  80  90 N/A 
Discrete           10  30  60  80  90  90

Singular metrics have only one value and no associated instance domain. Some metrics contain a set of values that share a common set of semantics for a specific instance, such as one value per processor, or one value per disk spindle, and so on.

---

Note: The PMDA implementation is solely responsible for choosing the instance identifiers that differentiate instances within the instance domain. The PMDA is also responsible for ensuring the uniqueness of instance identifiers in any instance domain.

---

  M[1]   M[2]   M[3] 
  M[4]   M[5]   M[6]

  M1[1]  M1[2]  M1[3] 
  M2[1]  M2[2]  M2[3]

typedef struct { 
    int         i_inst;         /* internal instance identifier */ 
    char        *i_name;        /* external instance identifier */ 
} pmdaInstid;

typedef struct { 
    pmInDom     it_indom;       /* indom, filled in */ 
    int         it_numinst;     /* number of instances */ 
    pmdaInstid  *it_set;        /* instance identifiers */ 
} pmdaIndom;

typedef struct { 
        int             pad:2; 
        unsigned int    domain:8;   /* the administrative PMD */ 
        unsigned int    serial:22;  /* unique within PMD */ 
} _pmInDom_int;

typedef struct { 
        unsigned int    serial:22;  /* unique within PMD */         
        unsigned int    domain:8;   /* the administrative PMD */ 
        int             pad:2;
} _pmInDom_int;

/* 
 * list of instances 
 */ 
static pmdaInstid color[] = {
    { 0, “red” }, { 1, “green” }, { 2, “blue” }
};
static pmdaInstid       *timenow = NULL;
static unsigned int     timesize = 0;
/*
 * list of instance domains
 */
static pmdaIndom indomtab[] = {
#define COLOR_INDOM     0
    { COLOR_INDOM, 3, color },
#define NOW_INDOM       1
    { NOW_INDOM, 0, NULL },
};
/*
 * all metrics supported in this PMDA - one table entry for each
 */
static pmdaMetric metrictab[] = {
/* numfetch */
    { NULL,
      { PMDA_PMID(0,0), PM_TYPE_U32, PM_INDOM_NULL, PM_SEM_INSTANT,
       PMDA_PMUNITS ( 0,0,0,0,0,0) }, },
/* color */
    { NULL,
      { PMDA_PMID(0,1), PM_TYPE_32, COLOR_INDOM, PM_SEM_INSTANT,
        PMDA_PMUNITS ( 0,0,0,0,0,0) }, },
/* time.user */
    { NULL,
      { PMDA_PMID(1,2), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
        PMDA_PMUNITS ( 0, 1, 0, 0, PM_TIME_SEC, 0 ) }, },
/* time.sys */
    { NULL,
      { PMDA_PMID(1,3), PM_TYPE_DOUBLE, PM_INDOM_NULL, PM_SEM_COUNTER,
        PMDA_PMUNITS ( 0, 1, 0, 0, PM_TIME_SEC, 0 ) }, },
/* now */
    { NULL,
      { PMDA_PMID(2,4), PM_TYPE_U32, NOW_INDOM, PM_SEM_INSTANT,
        PMDA_PMUNITS { 0,0,0,0,0,0 } }, },
};

dp->version.two.fetch = simple_fetch;
dp->version.two.instance = simple_instance;

A suggested approach to writing a PMDA is to write a standalone program to extract the values from the target domain and then incorporate this program into the PMDA framework. This approach avoids concurrent debugging of two distinct problems:

These are some possible ways of exporting the data from the target domain:

Most of these approaches require some further data processing by the PMDA.

The PCP protocols expect PMDAs to return the current values for performance metrics when requested, and with short delay (low latency). For some target domains, access to the underlying instrumentation may be costly or involve unpredictable delays (for example, if the real performance data is stored on some remote host or network device). In these cases, it may be necessary to separate probing for new performance data from servicing PMCD requests.

An architecture that has been used successfully for several PMDAs is to create one or more sproc child processes to obtain information while the main process communicates with PMCD; see the sproc(2) man page.

---

Note: The sproc(2) function is not available on Linux. You can use the exec(2)/fork(2) and setgpid(2) calls as an alternative.

---

At the simplest deployment of this arrangement, the two processes may execute without synchronization. Pthreads have also been used as a more portable multithreading mechanism; see the pthreads(5) man page.

By contrast, a complex deployment would be one in which the refreshing of the metric values must be atomic, and this may require double buffering of the data structures. It also requires coordination between parent and child processes.

---

Tip: Since PMAPI is not thread-safe, only one PMDA process or thread of control should call any PMAPI functions, and this would typically be the thread servicing requests from the PMCD.

---

One caveat about this style of caching PMDA--it is generally better if the PMDA converts counts to rates based upon consecutive periodic sampling from the underlying instrumentation. By exporting precomputed rate metrics with instantaneous semantics, the PMDA prevents the PCP monitor tools from computing their own rates upon consecutive PMCD fetches (which are likely to return identical values from a caching PMDA).

The PMNS file defines the name space of the PMDA. It is a simple text file that is used during installation to expand the Name Space of the PMCD process. The format of this file is described by the pmns(4) man page.

Client processes will not be able to access the PMDA metrics if the PMNS file is not installed as part of the PMDA installation procedure on the collector host. The installed list of metric names and their corresponding PMIDs can be found in /var/pcp/pmns/root.

Example 2-15 shows the simple PMDA, which has five metrics:

simple {
    numfetch    SIMPLE:0:0
    color       SIMPLE:0:1
    time
    now         SIMPLE:2:4
}
simple.time {
    user        SIMPLE:1:2
    sys         SIMPLE:1:3
}

Metrics that have different clusters do not have to be specified in different subtrees of the PMNS. Example 2-16 shows an alternative PMNS for the simple PMDA:

simple { 
    numfetch    SIMPLE:0:0 
    color       SIMPLE:0:1 
    usertime    SIMPLE:1:2 
    systime     SIMPLE:1:3 
}

For each metric defined within a PMDA, the PMDA developer is strongly encouraged to provide both terse and extended help text to describe the metric, and perhaps provide hints about the expected value ranges.

The help text is used to describe each metric in the visualization tools and pminfo with the -T option. The help text, such as the help text for the simple PMDA in Example 2-17, is specified in a specially formatted file, normally called help. This file is converted to the expected run-time format using the newhelp command; see the newhelp(1) man page. Converted help text files are usually placed in the PMDA's directory below /var/pcp/pmdas as part of the PMDA installation procedure.

@ SIMPLE.1 Instance domain “colour” for simple PMDA
Universally 3 instances, “red” (0), “green” (1) and “blue” (3).

@ SIMPLE.2 Dynamic instance domain “time” for simple PMDA
An instance domain is computed on-the-fly for exporting current time
information. Refer to the help text for simple.now for more details.

@ simple.numfetch Number of pmFetch operations.
The cumulative number of pmFetch operations directed to “simple” PMDA.

This counter may be modified with pmstore(1).

@ simple.color Metrics which increment with each fetch
This metric has 3 instances, designated “red”, “green” and “blue”.

The value of the metric is monotonic increasing in the range 0 to
255, then back to 0.  The different instances have different starting
values, namely 0 (red), 100 (green) and 200 (blue).

The metric values my be altered using pmstore(1).

@ simple.time.user Time agent has spent executing user code
The time in seconds that the CPU has spent executing agent user code.

@ simple.time.sys Time agent has spent executing system code
The time in seconds that the CPU has spent executing agent system code.

@ simple.now Time of day with a configurable instance domain
The value reflects the current time of day through a dynamically
reconfigurable instance domain.  On each metric value fetch request,
the agent checks to see whether the configuration file in
/var/pcp/pmdas/simple/simple.conf has been modified - if it has then
the file is re-parsed and the instance domain for this metric is again
constructed according to its contents.

This configuration file contains a single line of comma-separated time
tokens from this set:
  “sec”  (seconds after the minute),
  “min”  (minutes after the hour),
  “hour” (hour since midnight).

An example configuration file could be:  sec,min,hour
and in this case the simple.now metric would export values for the
three instances “sec”, “min” and “hour” corresponding respectively to
the components seconds, minutes and hours of the current time of day.

The instance domain reflects each token present in the file, and the
values reflect the time at which the PMDA processes the fetch.

Evolution of a PMDA, or more particularly the underlying instrumentation to which it provides access, over time naturally results in the appearance of new metrics and the disappearance of old metrics. This creates potential problems for PMAPI clients and PCP tools that may be required to interact with both new and former versions of the PMDA.

The following guidelines are intended to help reduce the complexity of implementing a PMDA in the face of evolutionary change, while maintaining predictability and semantic coherence for tools using the PMAPI, and for end users of those tools.

Both daemon and DSO PMDAs must handle multiple request types from PMCD. A daemon PMDA communicates with PMCD using the PDU protocol, while a DSO PMDA defines callbacks for each request type. To avoid duplicating this PDU processing (in the case of a PMDA that can be installed either as a daemon or as a DSO), and to allow a consistent framework, pmdaMain can be used by a daemon PMDA as a wrapper to handle the communication protocol using the same callbacks as a DSO PMDA. This allows a PMDA to be built as both a daemon and a DSO, and then to be installed as either.

To further simplify matters, default callbacks are declared in /usr/include/pcp/pmda.h:

Each callback takes a pmdaExt structure as its last argument. This structure contains all the information that is required by the default callbacks in most cases. The one exception is pmdaFetch, which needs an additional callback to instantiate the current value for each supported combination of a performance metric and an instance.

Therefore, for most PMDAs all the communication with PMCD is automatically handled by functions in libpcp.so and libpcp_pmda.so.

static int
trivial_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
{
   __pmID_int          *idp = (__pmID_int *)&(mdesc->m_desc.pmid);
   if (idp->cluster != 0 || idp->item != 0)
       return PM_ERR_PMID;
   else if (inst != PM_IN_NULL)
       return PM_ERR_INST;
   atom->l = time(NULL);
   return 0;
}

static int
simple_fetch(int numpmid, pmID pmidlist[], pmResult **resp, pmdaExt *pmda)
{
    numfetch++;
    simple_timenow_check();
    simple_timenow_refresh();
    return pmdaFetch(numpmid, pmidlist, resp, pmda);
}

static int
simple_fetchCallBack(pmdaMetric *mdesc, unsigned int inst, pmAtomValue *atom)
{
   int                 i;
   static int          oldfetch = 0;
   static struct tms   tms;
   __pmID_int          *idp = (__pmID_int *)&(mdesc->m_desc.pmid);
   if (inst != PM_IN_NULL &&
       !(idp->cluster == 0 && idp->item == 1) &&
       !(idp->cluster == 2 && idp->item == 4))
       return PM_ERR_INST;
   if (idp->cluster == 0) {
       if (idp->item == 0) {                   /* simple.numfetch */
           atom->l = numfetch;
       }

       else if (idp->item == 1) {              /* simple.color */
            switch (inst) {
            case 0:                             /* red */
                red = (red + 1) % 256;
                atom->l = red;
                break;
            case 1:                             /* green */
                green = (green + 1) % 256;
                atom->l = green;
                break;
            case 2:                             /* blue */
                blue = (blue + 1) % 256;
                atom->l = blue;
                break;
            default:
                return PM_ERR_INST;
            }
       }
       else
           return PM_ERR_PMID;

   else if (idp->cluster == 1) {               /* simple.time */
       if (oldfetch < numfetch) {
           times(&tms);
           oldfetch = numfetch;
       }
       if (idp->item == 2)                     /* simple.time.user */
           atom->d = (tms.tms_utime / (double)CLK_TCK);
       else if (idp->item == 3)                /* simple.time.sys */
           atom->d = (tms.tms_stime / (double)CLK_TCK);
       else
           return PM_ERR_PMID;
    }

    else if (idp->cluster == 2) {
        if (idp->item == 4) {                 /* simple.now */
            /* this loop will always match one of the named */
            /* time constants from the timeslices structure */
            for (i = 0; i < num_timeslices; i++) {
                if (inst == timeslices[i].inst_id) {
                    atom->l = timeslices[i].tm_field;
                    break;
                }
            }
            if (i == num_timeslices)
                return PM_ERR_INST;
        }
        else 
            return PM_ERR_PMID;
    }

static int
simple_store(pmResult *result, pmdaExt *pmda)
{
   int         i, j, val, sts = 0;
   pmAtomValue av;
   pmValueSet  *vsp = NULL;
   __pmID_int  *pmidp = NULL;
   for (i = 0; i < result->numpmid; i++) {
       vsp = result->vset[i];
       pmidp = (__pmID_int *)&vsp->pmid;
       if (pmidp->cluster == 0) {  /* storable metrics are cluster0 */
           switch (pmidp->item) {
               case 0:                           /* simple.numfetch */
                   val = vsp->vlist[0].value.lval;
                   if (val < 0) {
                       sts = PM_ERR_SIGN;
                       val = 0;
                   }
                   numfetch = val;
                   break;
               case 1:                             /* simple.color */
                   for (j = 0; j < vsp->numval && sts == 0; j++) {
                       val = vsp->vlist[j].value.lval;
                       if (val < 0) {
                           sts = PM_ERR_SIGN;
                           val = 0;
                       } if (val > 255) {
                           sts = PM_ERR_CONV;
                           val = 255;
                       }

                       switch (vsp->vlist[j].inst) {
                           case 0:                         /* red */
                               red = val;
                               break;
                           case 1:                         /* green */
                               green = val;
                               break;
                           case 2:                         /* blue */
                               blue = val;
                               break;
                           default:
                               sts = PM_ERR_INST;
                       }

               default:
                   sts = PM_ERR_PMID;
                   break;
           }
       }

       else if ((pmidp->cluster == 1 &&
                (pmidp->item == 2 || pmidp->item == 3)) ||
                (pmidp->cluster == 2 && pmidp->item == 4)) {
           sts = -EACCES;
           break;
       }
       else {
           sts = PM_ERR_PMID;
           break;
       }
   }
   return sts;
}

PMDA structures used with the pcp_pmda library are defined in /usr/include/pcp/pmda.h. Example 2-29 and Example 2-30 describe the pmdaInterface and pmdaExt structures.

typedef struct {
   int domain;     /* set/return performance metrics domain id here */
   struct {
       unsigned int pmda_interface : 8;  /* PMDA DSO version */
       unsigned int pmapi_version : 8;   /* PMAPI version */
       unsigned int flags : 16;          /* usage TBD */
   } comm;             /* set/return communication and version info */
   int status;         /* return initialization status here */
   union {
/* 
* Interface Version 2 (PCP 2.0) or later 
* PMDA_INTERFACE2, PMDA_INTERFACE3, ...
*/ 
       struct {
           pmdaExt *ext;
           int     (*profile)(__pmProfile *, pmdaExt *);
           int     (*fetch)(int, pmID *, pmResult **, pmdaExt *);
           int     (*desc)(pmID, pmDesc *, pmdaExt *);
           int     (*instance)(pmInDom, int, char *, __pmInResult **,
                   pmdaExt *);
           int     (*text)(int, int, char **, pmdaExt *);
           int     (*store)(pmResult *, pmdaExt *);
       } two;
   } version;
} pmdainterface;

typedef struct {
   unsigned int e_flags;       /* used internally within libpcp_pmda */
   void        *e_ext;         /* used internally within libpcp_pmda */
   char        *e_sockname;    /* socket name to pmcd */
   char        *e_name;        /* name of this pmda */
   char        *e_logfile;     /* path to log file */
   char        *e_helptext;    /* path to help text */
   int         e_status;       /* =0 is OK */
   int         e_infd;         /* input file descriptor from pmcd */
   int         e_outfd;        /* output file descriptor to pmcd */
   int         e_port;         /* port to pmcd */
   int         e_singular;     /* =0 for singular values */
   int         e_ordinal;      /* >=0 for non-singular values */
   int         e_direct;       /* =1 if pmid map to meta table */
   int         e_domain;       /* metrics domain */
   int         e_nmetrics;     /* number of metrics */
   int         e_nindoms;      /* number of instance domains */
   int         e_help;         /* help text comes via this handle */
   __pmProfile *e_prof;        /* last received profile */
   pmdaIoType  e_io;           /* connection type to pmcd */
   pmdaIndom   *e_indoms;      /* instance domain table */
   pmdaIndom   *e_idp;         /* instance domain expansion */
   pmdaMetric  *e_metrics;     /* metric description table */
   pmdaResultCallBack e_resultCallBack; /* to clean up pmResult after fetch */
   pmdaFetchCallBack  e_fetchCallBack;  /* to assign metric values in fetch */
   pmdaCheckCallBack  e_checkCallBack;  /* callback on receipt of a PDU */
   pmdaDoneCallBack   e_doneCallBack;   /* callback after PDU is processed */
} pmdaExt;

The initialization process for a PMDA involves opening help text files, assigning callback function pointers, adjusting the metric and instance identifiers to the correct domains, and much more. The initialization of a daemon PMDA also differs significantly from a DSO PMDA, since the pmdaInterface structure is initialized by main or the PMCD process, respectively.

As described in Section 2.2.2, an initialization function is provided by a DSO PMDA and called by PMCD. Using the standard PMDA wrappers, the same function can also be used as part of the daemon PMDA initialization. This PMDA initialization function performs the following tasks:

If the PMDA uses the common data structures defined for the pcp_pmda library, most of these requirements can be handled by the default pmdaInit function; see the pmdaInit(3) man page.

Because the initialization function is the only initialization opportunity for a DSO PMDA, the common initialization function should also perform any DSO-specific functions that are required. A default implementation of this functionality is provided by the pmdaDSO function; see the pmdaDSO(3) man page.

void trivial_init(pmdaInterface *dp)
{
   pmdaSetFetchCallBack(dp, trivial_fetchCallBack);
   pmdaInit(dp, NULL, 0,
            metrictab, sizeof(metrictab)/sizeof(metrictab[0]));
}

static int      isDSO = 1;              /* =0 I am a daemon */
void simple_init(pmdaInterface *dp)
{
   if (isDSO)
       pmdaDSO(dp, PMDA_INTERFACE_2, “simple DSO”,
               “/var/pcp/pmdas/simple/help”);
   if (dp->status != 0)
       return;
   dp->version.two.fetch = simple_fetch;
   dp->version.two.store = simple_store;
   dp->version.two.instance = simple_instance;
   pmdaSetFetchCallBack(dp, simple_fetchCallBack);
   pmdaInit(dp, indomtab, sizeof(indomtab)/sizeof(indomtab[0]),
            metrictab, sizeof(metrictab)/sizeof(metrictab[0]));
}

In addition to the initialization function that can be shared by a DSO and a daemon PMDA, a daemon PMDA must also meet the following requirements:

All these requirements can be handled by default initialization functions in the pcp_pmda library; see the pmdaDaemon(3), pmdaGetOpt(3), pmdaOpenLog(3), pmdaConnect(3), and pmdaMain(3) man pages.

The simple PMDA requires no additional command-line arguments other than those handled by pmdaGetOpt as shown in Example 2-33. For additional information, see the pmdaGetOpt(3) man page.

int
main(int argc, char **argv)
{
   int                 err = 0;
   pmdaInterface       dispatch;
   char                *p;
   /* trim cmd name of leading directory components */
   pmProgname = argv[0];
   for (p = pmProgname; *p; p++) {
       if (*p == `/')
           pmProgname = p+1;
   }
   isDSO = 0;
   pmdaDaemon(&dispatch, PMDA_INTERFACE_2, pmProgname, SIMPLE,
              “simple.log”, “/var/pcp/pmdas/simple/help”);
   if (pmdaGetOpt(argc, argv, “D:d:i:l:pu:?”, &dispatch, &err)) != EOF)
       err++;
   if (err)
       usage();
   pmdaOpenLog(&dispatch);
   simple_init(&dispatch);
   simple_timenow_check();
   pmdaConnect(&dispatch);
   pmdaMain(&dispatch);
   exit(0);
   /*NOTREACHED*/
}

Thoroughly testing a PMDA with PMCD is difficult, although testing a daemon PMDA is marginally simpler than testing a DSO PMDA. If a DSO PMDA exits, PMCD also exits because they share a single address space and control thread. If the PMDA dumps core, dbx and related tools on IRIX or gdb(1) on Linux cannot reasonably explore the generated core image, which includes the PMCD image and any other active DSO PMDAs. For more information, see the dbx(1) or gdb(1)man page.

The difficulty in using PMCD to test a daemon PMDA results from PMCD requiring timely replies from the PMDA in response to request PDUs. Although a timeout period can be set in /etc/config/pmcd.options on IRIX or /var/pcp/config/pmcd/pmcd.options on Linux, attaching dbx or gdb to the PMDA process (or any other long delay) might cause an already running PMCD to close its connection with the PMDA. If timeouts are disabled, PMCD could wait forever to connect with the PMDA.

If you suspect a PMDA has been terminated due to a timeout failure, check the PMCD log file, usually /var/adm/pcplog/pmcd.log for IRIX or /var/log/pcp/pmcd/pmcd.log for Linux.

A more robust way of testing a PMDA is to use the dbpmda tool, which is similar to PMCD except that dbpmda provides complete control over the PDUs that are sent to the PMDA, and there are no time limits--it is essentially an interactive debugger for exercising a PMDA. See the dbpmda(3) man page for details.

In addition, careful use of PCP debugging flags can produce useful information concerning a PMDA's behavior; see the pmapi(3) and pmdbg(1) man pages for a discussion of the PCP debugging and tracing framework.

You can activate debugging flags in PMCD and most other PCP tools with the -D command-line option. Supported flags can be listed with the pmdbg command; see the pmdbg(1) man page. Setting the debug flag for PMCD in /etc/config/pmcd.options on IRIX or /var/pcp/config/pmcd/pmcd.options on Linux might generate too much information to be useful, especially if there are other clients and PMDAs connected to the PMCD process.

The PMCD debugging flag can also be changed dynamically by storing a new value into the metric pmcd.control.debug:

# pmstore pmcd.control.debug 5

Most of the pcp_pmda library functions log additional information if the DBG_TRACE_LIBPMDA flag is set within the PMDA; see the pmda(3) man page. The command-line argument -D is trapped by pmdaGetOpt to set the global debugging control variable pmDebug. Adding tests within the PMDA for the DBG_TRACE_APPL0, DBG_TRACE_APPL1, and DBG_TRACE_APPL2 trace flags permits different levels of information to be logged to the PMDA's log file.

All diagnostic, debugging, and tracing output from a PMDA should be written to the standard error stream. By convention, all debugging information is enclosed by preprocessor #ifdef DEBUG statements so that they can be compiled out of the program at a later stage, if required.

Adding this segment of code to the simple_store metric causes a log message to be sent to the current log file whenever pmstore attempts to change simple.numfetch and pmDebug has the DBG_TRACE_APPL0 flag set as shown in Example 2-34:

   case 0: /* simple.numfetch */ 
       val = vsp->vlist[0].value.lval; 
       if (val < 0) { 
           sts = PM_ERR_SIGN; 
           val = 0; 
       } 
#ifdef DEBUG 
       if (pmDebug & DBG_TRACE_APPL0) { 
           fprintf(stderr,  
                "simple: %d stored into numfetch", val); 
       } 
#endif 
       numfetch = val; 
       break;

For a description of pmstore, see the pmstore(1) man page.

The dbpmda utility provides a simple interface to the PDU communication protocol. It allows daemon and DSO PMDAs to be tested with most request types, while the PMDA process may be monitored with dbx, par on IRIX or gdb, strace on Linux, and other diagnostic tools. The dbpmda(1) man page contains a sample session with the simple PMDA.

A PMDA is fully installed when these tasks are completed:

The Makefile should include an install target to compile and link the PMDA (as a DSO, or a daemon or both) in the PMDA directory, and in the case of a DSO PMDA, install the shared library in /var/pcp/lib for IRIX and /usr/share/pcp/lib for Linux. The clobber target should remove any files created as a by-product of the install target.

You may wish to use /var/pcp/pmdas/simple/Makefile as a template for constructing a new PMDA Makefile; changing the assignment of IAM from simple to newbie would account for most of the required changes.

Since the object format of a DSO PMDA must match the object format of PMCD, which in turn must match the object format of the booted operating system kernel, there might be multiple DSO targets in the Makefile. For an example on IRIX, see targets mips_o32.pmda_$(IAM).so, mips_n32.pmda_$(IAM).so, and mips_64.pmda_$(IAM).so for the simple PMDA.

The Install script should make use of the generic procedures defined in the script /usr/pcp/lib/pmdaproc.sh on IRIX and /usr/share/pcp/lib/pmdaproc.sh on Linux, and may be as straightforward as the one used for the trivial PMDA, shown in Example 2-35:

# Get the common procedures and variable assignments
#
. /usr/pcp/lib/pmdaproc.sh
# The name of the PMDA
#
iam=trivial
# Do it
#
pmdaSetup
pmdainstall
exit 0

The variables, shown in Table 2-1, may be assigned values to modify the behavior of the pmdaSetup and pmdainstall procedures from /usr/pcp/lib/pmdaproc.sh or /usr/share/pcp/lib/pmdaproc.sh on Linux.

In addition, the variables do_pmda and do_check will be set to reflect the intention to install the PMDA (as opposed to install just the PMNS) and to check the availability of the metrics once the PMDA is installed. By default, each variable is true; however, the command-line options -N and -Q to Install may be used to set the variables to false, as follows: do_pmda (-N) and do_check (-N or -Q).

The variables may also have their assignments changed by the user's response to the common prompt as shown in Example 2-36:

You will need to choose an appropriate configuration for installation 
of the ... Performance Metrics Domain Agent (PMDA).
  collector   collect performance statistics on this system
  monitor     allow this system to monitor local and/or remote systems
  both        collector and monitor configuration for this system

Obviously, for anything but the most trivial PMDA, after calling the pmdaSetup procedure, the Install script should also prompt for any PMDA-specific parameters, which are typically accumulated in the args variable and used by the pmdainstall procedure.

The detailed operation of the pmdainstall procedure involves the following tasks:

There are some PMDA changes that may trick PMCD into thinking nothing has changed, and not restarting the PMDA. Most notable are changes to the PMDA executable. In these cases, you may need to explicitly remove the PMDA as described in Section 2.8.3, or more drastically, restart PMCD on IRIX as follows:

# /etc/init.d/pcp start

Restart PMCD on Linux as follows:

# /etc/rc.d/init.d/pcp start

The files /var/pcp/pmdas/*/Install provide a wealth of examples that may be used to construct a new PMDAInstall script.

When invoked with a -N command-line option, the PMDA Install script may be used to update the PMNS without installing the PMDA. This is typically used on a monitoring system to populate the local PMNS with the names of the performance metrics from a PMDA installed on a remote host. The -N option also installs pmchart views useful on a monitoring system.

The simplest way to stop a PMDA from running, apart from killing the process, is to remove the entry from /etc/pmcd.conf and signal PMCD (with SIGHUP) to reread its configuration file. To completely remove a PMDA requires the reverse process of the installation, including an update of the Performance Metrics Name Space (PMNS).

This typically involves a Remove script in the PMDA directory that uses the same common procedures as the Install script described Section 2.8.1.

The /var/pcp/pmdas/*/Remove files provide a wealth of examples that may be used to construct a new PMDA Remove script.

Most PCP tools have their own configuration file format for specifying which metrics to view or to log. By using canned configuration files that monitor key metrics of the new PMDA, users can quickly see the performance of the target system, as characterized by key metrics in the new PMDA.

Any configuration files that are created should be kept with the PMDA and installed into the appropriate directories when the PMDA is installed.

As with all PCP customization, some of the most valuable tools can be created by defining views, scenes, and control-panel layouts that combine related performance metrics from multiple PMDAs or multiple hosts.

Parameterized alarm configurations can be created using the pmieconf facilities; see the pmieconf(1) and pmie(1) man pages. In addition, pmie rules involving metrics from the new PMDA may be created directly.

Daily logs can be specified in pmlogger configuration files, or with the pmlogger_daily mechanism; see the pmlogger(1) and pmlogger_daily(1) man pages. The services of pmsnap may be used to incorporate the new performance metrics into charts that may be periodically regenerated and published via a World Wide Web server.