grioqos [options] [streamID|fs] [delay [count]]

-c	Clears the screen before printing each new set of statistics.
-h	Prints a usage message.
-i	Reports the following low-level QoS metrics for all currently configured sampling intervals: minbw maxbw lastbw minio maxio lastio For details about these metrics, see “Quality-of-Service Metrics”.
-I intervals	Reports the same low-level QoS metrics as -i , but for a specified range of sampling intervals. intervals is a comma-separated list of sampling intervals expressed as either a number of I/Os or a time interval in msecs. For example, the following would report results averaged over the last 5 and 10 I/Os, and over the last 1 and 2 seconds, respectively: -I 5,10,1000ms,2000ms
-l	Lists active streams in an easily parsed form (one per line with the following fields: Filesystem mount point (or the string <unmounted> if the filesystem is not mounted) Type of the stream Stream ID Reserved bandwidth reported in bytes and msecs Process ID for application-created streams
-m	Enables monitoring mode, which reports the following high-level stream and I/O metrics: bytes msecs bckt bckt (max) total rate bklg issd idle thrt wait See “Monitoring Stream and I/O Metrics” for more information.
-n	Prints a more human-readable version of the performance information reported with the -i option for all currently configured sampling intervals: lastbw minbw maxio For more information, see “Quality-of-Service Metrics” The minimum bandwidth and maximum average service time are the metrics of most concern when attempting to deliver guaranteed data rates.
-N intervals	Reports the same metrics as -n, but for a specified range of sampling intervals. intervals is a comma-separated list of sampling intervals expressed as either a number of I/Os or a time interval in msecs. For example, the following would report results averaged over the last 5 and 10 I/Os, and over the last 1 and 2 seconds, respectively: -N 5,10,1000ms,2000ms
-o file	Logs output to the specified file.
-r	Resets the specified statistics when used with one of the following options: High-level stream statistics: -m Low-level QoS statistics: -i, -I, -n, -N, -t, or -T The -r option is ignored if none of these other options is specified. GRIO will continue to update some kernel statistics even when no I/O is being performed (such as when the rate metric reported in the -m mode is updated even on an idle stream). In order to get results that accurately correspond with those seen by a user application, you should start grioqos with the -r option at the same time that the application test begins.
-R intervals	Reconfigures the kernel QoS monitoring intervals and resets the statistics. This allows you to change the set of sampling intervals used in the kernel to compute recent bandwidth and average service time. intervals is a comma-separated list of sampling intervals expressed as either a number of I/Os or a time interval in msecs. For example, the following would report results averaged over the last 5 and 10 I/Os, and over the last 1 and 2 seconds, respectively: -R 5,10,1000ms,2000ms By default, GRIO is configured to compute statistics for a wide range of sampling intervals. However, it can be useful to change these intervals using the -R option when a monitored application has a buffering behavior that is not well-matched by the default intervals. Note: GRIO always configures two additional intervals automatically: The single sample interval, which tracks the best and worst case service times for individual I/Os The maximum interval, which is as large as the kernel data structures can accommodate
-s	Prints a more human-readable summary of active streams than -l. Results are grouped per filesystem and include the following: Stream type Process ID (for application streams) Bandwidth reservation in MB/s Stream IDs (when -v is also specified) For more information about the output format, see the grioqos(1M) man page.
-t	Displays a per-stream I/O service time histogram for all buckets.
-T buckets	Displays a per-stream I/O service time histogram for the specified buckets. You can display a ranges of values. For example, the following would cause the values of 11 histogram buckets to be displayed: -T 0,1,2,3,20-25,52
-v	Display verbose output (used with -s).
streamID	Specifies the ID of an active GRIO stream.
fs	Specifies a path that identifies a mounted GRIO-managed filesystem.
delay	Specifies the length of time in seconds that grioqos should sleep before retrieving each new set of statistics.
count	Specifies the total number of samples to be retrieved.

If you specify grioqos without any arguments, it prints a usage message by default.

To terminate grioqos, press Ctrl-C on Windows or send a SIGINT on other platforms.

Interpreting the statistics collected by grioqos requires a basic understanding of the GRIO scheduler.

GRIO uses the token bucket abstraction to limit the average rate and burstiness of I/O flowing to or from the filesystem. Conceptually, each stream has a bucket of tokens. Each token confers the right to issue one unit of I/O. Tokens are added to the token bucket at a rate corresponding to the GRIO reservation and accumulate up to the maximum size of the bucket, at which point any further tokens are discarded. When a new I/O request arrives, it is issued if there are sufficient tokens available to the token bucket; if there are insufficient tokens, it is added to the throttle queue for the stream, where it is held for a short period before the token bucket is checked again. The rate at which tokens accrue to the token bucket controls the average rate of the stream. The maximum size of the token bucket controls the size of the largest burst of I/O that can be issued.

The ability to issue a temporary burst of I/O above the reserved data rate is important. It is the mechanism within GRIO by which an application or device that temporarily falls below the required data rate can catch up, thus preserving the required average data rate.

GRIO implements a variation of the weighted round-robin scheduling discipline. At each scheduler activation, it visits each stream in the system and issues as much I/O as it can, up to the limit of the token bucket. The order in which the streams are visited is always the same. To increase the determinism of the resulting I/O flow, GRIO will (on platforms where it is possible) attempt to disable further I/O reordering operations in lower-level devices.

In monitoring mode (enabled with -m), grioqos reports the following metrics:

The stream utilization metrics (idle, thrt, and wait) can be useful when trying to understand the interaction between an application, the GRIO scheduler, and the storage device. Table 5-1 describes commonly observed behaviors and their corresponding metrics.

Depending on the amount of I/O buffering an application performs, it may be more or less sensitive to variation in I/O service time, also known as jitter. This can vary from tens of seconds for applications that have large buffers and use threaded or asynchronous I/O, to tens of milliseconds for single-threaded applications with little buffering that require a low upper bound on I/O service time.

Approaches to measuring I/O performance often tend to focus at the ends of this spectrum, measuring one of the following (which can be limiting as a result):

The GRIO QoS infrastructure provides a configurable mechanism for monitoring performance over the entire range of time scales from the service times of individual I/Os to the sustained bandwidth over long sampling intervals. It can do so for an individual application or over a period of time without instrumenting or otherwise disrupting the performance of the application.

Within the kernel, GRIO records the I/O completion times for all recent I/Os to or from a stream. From this high-resolution data, it computes a number of derived metrics that can be efficiently exported to userspace. You can change the monitoring intervals over which these metrics are computed by using grioqos. Sampling intervals can be expressed as either a time t (such as 1000ms ) or as a number of individual samples n. For instance, grioqos can display average I/O service time and bandwidth for the last four I/Os, the last 200ms, the last second, and so forth.

GRIO computes the following metrics for each configured sampling interval:

There is a size restriction on the kernel structures used to hold recent I/O statistics. If a requested metric cannot be computed because there is insufficient data, a single hyphen (-) is printed. This can also happen when the QoS metrics have been recently reset using the -r or -R options. For example, requesting a sampling interval of 10000ms may display only a hyphen (-) because the GRIO kernel structures cannot hold enough individual samples to compute an average over ten seconds. However, for most I/O rates and sampling intervals, the kernel structures should be adequate.

Use care when interpreting the low-level QoS statistics. A number of the bandwidth and service time measures only make sense if they have been recorded during a period of continuous, consistent application I/O (for example, for a video playout).

The lastbw and maxbw metrics are meaningful regardless of the behavior of the application. However, minbw tracks all interruptions to the flow of I/O. This includes interruptions due to the normal operation of the application as opposed to an actual service interruption in the filesystem or device. Thus, if the application stops and starts I/O during the sampling period, this will be recorded in the minbw, which will in turn be of little use in detecting a real service interruption and is unlikely to provide any useful insight into the performance of the application and system.

Similarly, the lastio metric is most useful if the application uses a consistent request size when issuing I/O to the filesystem. If the application issues I/O of widely varying size, then the service time is permuted both by filesystem and device issues and the behavior of the application. For such applications, this makes it very difficult to determine the origin of a performance issue. This is particularly true for non-GRIO streams., which manage all of the I/O on a node that does not otherwise have an explicit GRIO reservation. This includes the following:

The result is that the non-GRIO stream may see a large variation in I/O sizes and the average service time of those I/Os is unlikely to provide useful insight into the performance of the system.

This section shows grioqos used to monitor a GRIO-aware application. High-level stream and low-level quality-of-service metrics are collected. The application is temporarily suspended to show the effect on the stream utilization and average data rate. The example filesystem /mirror has a qualified bandwidth of 30 MB/s.