This chapter describes the performance monitoring tools available in Performance Co-Pilot (PCP). This product provides a group of commands and tools for measuring system performance. Each tool is described completely by its own man page. The man pages are accessible through the man command. For example, the man page for the tool pmdumptext is viewed by entering the following command:
man pmdumptext |
The following major sections are covered in this chapter:
Section 4.1, discusses pmkstat, a utility that provides a periodic one-line summary of system performance.
Section 4.2, discusses pmdumptext , a utility that shows the current values for named performance metrics.
Section 4.3, describes pmval , a utility that displays performance metrics in ASCII tables.
Section 4.4, describes pminfo , a utility that displays information about performance metrics.
Section 4.5, describes the use of the pmstore utility to arbitrarily set or reset selected performance metric values.
The following sections describe the various graphical and text-based PCP tools used to monitor local or remote system performance.
The pmkstat command provides a periodic, one-line summary of system performance. This command is intended to monitor system performance at the highest level, after which other tools may be used for examining subsystems to observe potential performance problems in greater detail. After entering the pmkstat command, you see output similar to the following, with successive lines appearing periodically:
pmkstat # hostname load avg: 0.26, interval: 5 sec, Thu Jan 19 12:30:13 2002 runq | memory | system | disks| cpu mem swp | free page | scall ctxsw intr| rd wr|usr sys idl wt 0 0 16268 0 64 19 2396 0 0 0 1 99 0 0 0 16264 0 142 45 2605 0 8 0 2 97 0 0 0 16268 0 308 62 2532 0 1 1 1 98 0 0 0 16268 0 423 88 2643 0 0 1 1 97 0 |
An additional line of output is added every five seconds. The update interval may be varied using the -t interval option.
The output from pmkstat is directed to standard output, and the columns in the report are interpreted as follows:
| runq | Average number of runnable processes in main memory ( mem) and in swap memory (swp) during the interval. | |
| memory | The free column indicates average free memory during the interval, in kilobytes. The page column is the average number of page-out operations per second during the interval. I/O operations caused by these page-out operations are included in the disk write I/O rate. | |
| system | System call rate (scall), context switch rate (ctxsw), and interrupt rate (intr). Rates are expressed as average operations per second during the interval. | |
| disks | Aggregated physical read (rd) and write (wr) rates over all disks, expressed as physical I/O operations issued per second during the interval. These rates are independent of the I/O block size. | |
| cpu | Percentage of CPU time spent executing user code ( usr), system and interrupt code (sys), idle loop (idl) and idle waiting for resources (wt), typically disk I/O. |
As with most PCP utilities, real-time metric, and archive logs are interchangeable.
For example, the following command uses the PCP archive log foo and the timezone of the host (tokyo) from which performance metrics in the archive were collected:
pmkstat -a foo -z Note: timezone set to local timezone of host "tokyo" # tokyo load avg: 1.06, interval: 5 sec, Thu Feb 2 08:42:55 2002 runq | memory | system | disks | cpu mem swp| free page| scall ctxsw intr| rd wr|usr sys idl wt 0 0 4316 0 195 64 2242 32 21 0 3 8 89 0 0 3976 0 279 86 2143 50 17 0 5 8 87 1 0 3448 0 186 63 2304 35 14 0 4 9 87 0 0 4364 0 254 81 2385 35 0 0 4 9 87 0 0 3696 0 266 92 2374 41 0 0 3 9 88 0 0 2668 42 237 81 2400 44 2 1 4 7 89 0 0 4644 100 206 68 2590 25 1 0 3 5 91 0 0 5384 0 174 63 2296 32 22 0 2 8 89 0 0 4736 0 189 65 2197 31 28 0 3 8 89 pmFetch: End of PCP archive log |
For complete information on pmkstat usage and command line options, see the pmkstat(1) man page.
The pmdumptext command displays performance metrics in ASCII tables, suitable for export into databases or report generators. It is a flexible command. For example, the following command provides continuous memory statistics on a host named serv:
pmdumptext -imu -h serv -f `%H:%M:%S' mem.util
Metric kernel fs_ctl _dirty _clean free user
Units b b b b b b
20:14:28 99.14M 6.03M 0.85M 98.42M 0.17G 0.16G |
See the pmdumptext(1) man page for more information.
The pmval command dumps the current values for the named performance metrics. For example, the following command reports the value of performance metric proc.nprocs once per second (by default), and produces output similar to this:
pmval proc.nprocs
etric: proc.nprocs
host: localhost
semantics: discrete instantaneous value
units: none
samples: all
interval: 1.00 sec
81
81
81
81
81 |
In this example, the number of running processes was reported once per second.
Where the semantics of the underlying performance metrics indicate that it would be sensible, pmval reports the rate of change or resource utilization.
For example, the following command reports idle processor utilization for each of four CPUs on the remote host moomba, each five seconds apart, producing output of this form:
pmval -h dove -t 5sec -s 4 kernel.percpu.cpu.idle
metric: kernel.percpu.cpu.idle
host: dove
semantics: cumulative counter (converting to rate)
units: millisec (converting to time utilization)
samples: 4
interval: 5.00 sec
cpu:1.1.0.a cpu:1.1.0.c cpu:1.1.1.a cpu:1.1.1.c
1.000 0.9998 0.9998 1.000
1.000 0.9998 0.9998 1.000
0.8989 0.9987 0.9997 0.9995
0.9568 0.9998 0.9996 1.000
|
Similarly, the following command reports disk I/O read rate every minute for just the disk /dev/dsk/dks0d1, and produces output similar to the following:
pmval -t 1min -i dks0d1 disk.dev.read
metric: disk.dev.read
host: localhost
semantics: cumulative counter (converting to rate)
units: count (converting to count / sec)
samples: indefinite
interval: 60.00 sec
dks0d1
33.67
48.71
52.33
11.33
2.333 |
The -r flag may be used to suppress the rate calculation (for metrics with counter semantics) and display the raw values of the metrics.
In the example below, manipulation of the time within the archive is achieved by the exchange of time control messages between pmval and pmtime.
pmval -g -a /var/log/pcp/pmlogger/myserver/960801 |
The pmval command is documented by the pmval(1) man page, and annotated examples of the use of pmval are in the PCP Tutorial.
The pminfo command displays various types of information about performance metrics available through the Performance Co-Pilot (PCP) facilities.
The -T option is extremely useful; it provides help text about performance metrics:
pminfo -T mem.util.fs_dirty mem.util.fs_dirty Help: The amount of memory in Kbytes that is holding file system data. |
The -t option displays the one-line help text associated with the selected metrics. The -T option prints more verbose help text.
Without any options, pminfo verifies that the specified metrics exist in the Name Space, and echoes those names. Metrics may be specified as arguments to pminfo using their full metric names. For example, this command returns the following response:
pminfo hinv.ncpu network.interface.total.bytes hinv.ncpu network.interface.total.bytes |
A group of related metrics in the Name Space may also be specified. For example, to list all of the hinv metrics you would use this command:
pminfo hinv hinv.physmem hinv.pagesize hinv.ncpu hinv.ndisk hinv.nfilesys hinv.machine hinv.map.scsi hinv.map.cpu_num hinv.map.cpu hinv.map.disk hinv.map.node hinv.map.router hinv.map.routerport hinv.map.xbow hinv.cpu.clock hinv.cpu.vendor hinv.cpu.model hinv.cpu.stepping hinv.cpu.cache hinv.cpu.bogomips hinv.nnode hinv.nrouter hinv.nrouterport hinv.nxbow hinv.interconnect |
If no metrics are specified, pminfo displays the entire collection of metrics. This can be useful for searching for metrics, when only part of the full name is known. For example, this command returns the following response:
pminfo | grep nfs nfs.client.calls nfs.client.reqs nfs.server.calls nfs.server.reqs nfs3.client.calls nfs3.client.reqs nfs3.server.calls nfs3.server.reqs |
The -d option causes pminfo to display descriptive information about metrics (refer to the pmLookupDesc(3) man page for an explanation of this metadata information). The following command and response show use of the -d option:
pminfo -d proc.nprocs disk.dev.read filesys.free
proc.nprocs
Data Type: 32-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
Semantics: discrete Units: none
disk.dev.read
Data Type: 32-bit unsigned int InDom: 60.1 0xf000001
Semantics: counter Units: count
filesys.free
Data Type: 64-bit unsigned int InDom: 60.5 0xf000005
Semantics: instant Units: Kbyte |
The -f option to pminfo forces the current value of each named metric to be fetched and printed. In the example below, all metrics in the group hinv are selected:
pminfo -f hinv
hinv.physmem
value 15701
hinv.pagesize
value 16384
hinv.ncpu
value 4
hinv.ndisk
value 6
hinv.nfilesys
value 2
hinv.machine
value "IP35"
hinv.map.scsi
No value(s) available!
hinv.map.cpu_num
inst [0 or "cpu:1.1.0.a"] value 0
inst [1 or "cpu:1.1.0.c"] value 1
inst [2 or "cpu:1.1.1.a"] value 2
inst [3 or "cpu:1.1.1.c"] value 3
hinv.map.cpu
inst [0 or "cpu:1.1.0.a"] value "/dev/hw/module/001c01/slab/0/node/cpubus/0/a"
inst [1 or "cpu:1.1.0.c"] value "/dev/hw/module/001c01/slab/0/node/cpubus/0/c"
inst [2 or "cpu:1.1.1.a"] value "/dev/hw/module/001c01/slab/1/node/cpubus/0/a"
inst [3 or "cpu:1.1.1.c"] value "/dev/hw/module/001c01/slab/1/node/cpubus/0/c"
hinv.map.disk
No value(s) available!
hinv.map.node
inst [0 or "node:1.1.0"] value "/dev/hw/module/001c01/slab/0/node"
inst [1 or "node:1.1.1"] value "/dev/hw/module/001c01/slab/1/node"
hinv.map.router
No value(s) available!
hinv.map.routerport
No value(s) available!
hinv.map.xbow
No value(s) available!
hinv.cpu.clock
inst [0 or "cpu:1.1.0.a"] value 800
inst [1 or "cpu:1.1.0.c"] value 800
inst [2 or "cpu:1.1.1.a"] value 800
inst [3 or "cpu:1.1.1.c"] value 800
hinv.cpu.vendor
inst [0 or "cpu:1.1.0.a"] value "GenuineIntel"
inst [1 or "cpu:1.1.0.c"] value "GenuineIntel"
inst [2 or "cpu:1.1.1.a"] value "GenuineIntel"
inst [3 or "cpu:1.1.1.c"] value "GenuineIntel"
hinv.cpu.model
inst [0 or "cpu:1.1.0.a"] value "0"
inst [1 or "cpu:1.1.0.c"] value "0"
inst [2 or "cpu:1.1.1.a"] value "0"
inst [3 or "cpu:1.1.1.c"] value "0"
hinv.cpu.stepping
inst [0 or "cpu:1.1.0.a"] value "6"
inst [1 or "cpu:1.1.0.c"] value "6"
inst [2 or "cpu:1.1.1.a"] value "6"
inst [3 or "cpu:1.1.1.c"] value "6"
hinv.cpu.cache
inst [0 or "cpu:1.1.0.a"] value 0
inst [1 or "cpu:1.1.0.c"] value 0
inst [2 or "cpu:1.1.1.a"] value 0
inst [3 or "cpu:1.1.1.c"] value 0
hinv.cpu.bogomips
inst [0 or "cpu:1.1.0.a"] value 1195.37
inst [1 or "cpu:1.1.0.c"] value 1195.37
inst [2 or "cpu:1.1.1.a"] value 1195.37
inst [3 or "cpu:1.1.1.c"] value 1195.37
hinv.nnode
value 2
hinv.nrouter
value 0
hinv.nrouterport
value 0
hinv.nxbow
Error: Functionality not yet implemented
hinv.interconnect
No value(s) available! |
The -h option directs pminfo to retrieve information from the specified host. If the metric has an instance domain, the value associated with each instance of the metric is printed:
pminfo -h dove.americas.sgi.com -f filesys.mountdir
filesys.mountdir
inst [0 or "/dev/xscsi/pci00.01.0/target81/lun0/part3"] value "/"
inst [1 or "/dev/xscsi/pci00.01.0/target81/lun0/part1"] value "/boot/efi" |
The -m option prints the Performance Metric Identifiers (PMIDs) of the selected metrics. This is useful for finding out which PMDA supplies the metric. For example, the output below identifies the PMDA supporting domain 4 (the leftmost part of the PMID) as the one supplying information for the metric environ.extrema.mintemp:
pminfo -m environ.extrema.mintemp environ.extrema.mintemp PMID: 4.0.3 |
The -v option verifies that metric definitions in the PMNS correspond with supported metrics, and checks that a value is available for the metric. Descriptions and values are fetched, but not printed. Only errors are reported.
Some instance domains are not enumerable. That is, it is not possible to ask for all of the instances at once. Only explicit instances may be fetched from such instance domains. This is because instances in such a domain may have a very short lifetime or the cost of obtaining all of the instances at once is very high. The proc metrics are an example of such an instance domain. The -f option is not able to fetch metrics with non-enumerable instance domains; however, the -F option tells pminfo to obtain a snapshot of all of the currently available instances in the instance domain and then to retrieve a value for each.
Complete information on the pminfo command is found in the pminfo(1) man page. There are examples of the use of pminfo in the PCP Tutorial.
From time to time you may wish to change the value of a particular metric. Some metrics are counters that may need to be reset, and some are simply control variables for agents that collect performance metrics. When you need to change the value of a metric for any reason, the command to use is pmstore.
 | Note: For obvious reasons, the ability to arbitrarily change the value of a performance metric is not supported. Rather, the PMCS selectively allows some metrics to be modified in a very controlled fashion. |
The basic syntax of the command is as follows:
pmstore metricname value |
There are also command line flags to further specify the action. For example, the -i option restricts the change to one or more instances of the performance metric.
The value may be in one of several forms, according to the following rules:
If the metric has an integer type, then value should consist of an optional leading hyphen, followed either by decimal digits or “0x” and some hexadecimal digits; “0X” is also acceptable instead of “0x.”
If the metric has a floating point type, then value should be in the form of an integer (described above), a fixed point number, or a number in scientific notation.
If the metric has a string type, then value is interpreted as a literal string of ASCII characters.
If the metric has an aggregate type, then an attempt is made to interpret value as an integer, a floating point number, or a string. In the first two cases, the minimal word length encoding is used; for example, “123” would be interpreted as a four-byte aggregate, and “0x100000000” would be interpreted as an eight-byte aggregate.
The following example illustrates the use of pmstore to enable performance metrics collection in the txmon PMDA (see /var/pcp/pmdas/txmon for the source code of the txmon PMDA). When the metric txmon.control.level has the value 0, no performance metrics are collected. Values greater than 0 enable progressively more verbose instrumentation.
pminfo -f txmon.count
txmon.count
No value(s) available!
pmstore txmon.control.level 1
txmon.control.level old value=0 new value=1
pminfo -f txmon.count
txmon.count
inst [0 or "ord-entry"] value 23
inst [1 or "ord-enq"] value 11
inst [2 or "ord-ship"] value 10
inst [3 or "part-recv"] value 3
inst [4 or "part-enq"] value 2
inst [5 or "part-used"] value 1
inst [6 or "b-o-m"] value 0 |
For complete information on pmstore usage and syntax, see the pmstore(1) man page.