This chapter explains how to start and stop the BDSpro service and how to configure its buffering and logging functions. It also explains the BDSpro services that are available to other applications.
This chapter contains the following section:
For detailed information about the BDSpro service, you can access the bds(1M) reference page by typing man bds.
The BDSpro service is ready to use after NFS is modified on the clients (see “Mounting Filesystems for BDSpro”) and the BDS setup is tested on the server (see “Verifying BDSpro Performance”). To start and stop the BDSpro service, enter these commands on the server:
/etc/init.d/BDSpro start
/etc/init.d/BDSpro stop
The start command above starts BDS automatically during the server startup sequence. This command reads a file called /etc/config/BDSpro.options, which you can create to specify BDS options, such as write buffering and buffer sizes.
For example, the following BDSpro.options file sets the logging level to 3, specifies write buffering, and sets single read and write buffering:
-loglevel 3 -writebehind 1 -nbufs 1 > /var/adm/bds.log 2>&1 |
To enable or disable the automatic starting of BDSpro during the server's startup sequential, enter these commands:
chkconfig BDSpro on
chkconfig BDSpro off
To turn on write buffering, use the -writebehind option in the BDSpro.options file (see the bds(1M) reference page and “Starting and Stopping the BDSpro Service”). On systems running IRIX 6.5 or higher, you can also specify write buffering from an application when the application opens a file.
If write buffering is in effect and an application must be certain that data is written to disk, the application should use the fsync system call (see the fsync(2) reference page for details). However, using fsync regularly compromises the benefits of write buffering.
By default, BDS allocates two memory buffers for each open file (see “BDS Buffer Management” for more information). However, you can reduce the buffer allocation to one buffer using the nbufs argument in a BDSpro.options file (see the sample file in “Starting and Stopping the BDSpro Service”).
For real-time filesystems, BDS sets buffer size to the filesystem extent size; for XLV logical volumes, BDSpro sets the size of read and write buffers to the size of the logical volume stripe (see “Tuning XLV Performance” in Chapter 2). If the filesystem is not striped, BDS sets the buffer size to the application's write or read size.
The amount of memory used in buffering is affected by the buffer size and the number of users accessing the server simultaneously. If you find that the default buffer size is requiring too much memory or unsuitable for other reasons, you can specify a buffer using any size by these methods:
Use the -buffersize argument in the /etc/config/BDSpro.options file (see the example in “Starting and Stopping the BDSpro Service” and the bds(1M) reference page for details).
Include the bdsbuffer= option on the mount command line, as shown in this example:
hip0-goliath:/ /bdsmnt -o bds,bdsbuffer=2097152,vers=3, rw 0 0
Buffer size specifications have this order of precedence:
The buffersize argument in the /etc/config/BDSpro.options file on the server.
The size specified by the application in fcntl (F_GETBDSATTR) (see “Using BDS from an Application”).
The bdsbuffer option in the mount command on the client.
The extent size of a real-time filesystem or the disk stripe size of an XLV logical volume.
The application read/write size.
This example sets the buffer size using a mount option on the client:
hip0-goliath:/ /bdsmnt -o bds, bdsauto=2000000, bdsbuffer=4194304, vers=3, rw 0 0 |
The entry in the previous example sets the buffer size to 4 MB. Because this parameter is set when the bdsmount filesystem is mounted, BDS will not use a calculated buffer size for bdsmount I/O; it will allocate buffers of 4 MB.
However, assume that this argument was used in the BDSpro.options file:
-buffersize 2097152
In this case, BDS uses a buffer size of 2 MB, overriding the 4 MB specified in the filesystem mount on the client.
You can disable BDS buffering by specifying a buffer size of zero (0). If you disable buffering, you can expect significant performance reductions. For example, laboratory tests with read-ahead buffering demonstrate a sustained performance of 87 MB per second, whereas without buffering, performance measured about 54 MB per second; with write-behind buffering, speeds of 93 MB per second dropped to 45 MB per second without buffering.
To run BDS over the ST protocol, the stp option must be specified on both the client and the server. To specify ST on the client, include the bdsproto=stp option in the /etc/fstab file or as an option on the mount command line. To specify ST on the server, include the -bdsproto stp option in the /etc/config/BDSpro.options file. If the stp option is not specified on both the client and the server, BDSpro attempts the transmission over TCP/IP by default. For more details on ST, see the ST web page at the following URL:
http://www.hippi.org/cST.html
BDSpro supports three levels of logging for errors and other events that occur in BDS transactions. These levels are options to the -loglevel argument to the BDS server. The levels are as follows:
Level 1 records the date and time of each event as well as an event description.
Level 2 records the date and time of each event, the process ID (PID) of the BDSpro event that reported it, and an event description.
Level 3 records the date and time of each event in milliseconds, the PID of the BDSpro process that reported it, and an event description.
For more information on turning on logging, see the bds(1) man page.
The list below contains all non-error events that are reported by the logging function.
(For level 1 logging) BANDWIDTH is reported when a client closes a file. It contains a brief summary of the number of bytes moved and the time elapsed between the file open and close:
BANDWIDTH client_1 moved 713.03 MB in 15.51 secs, 45.96 MB/sec
Levels 2 and higher add OPEN and CLOSE, which are reported when a file is opened and closed. OPEN contains the NFS version number and the device and inode number of the file; CLOSE contains the total number of requests sent between the client and server, the number of reads and the number of writes in this total, and the total number of disk I/Os made to process these requests. CLOSE is always reported before BANDWIDTH:
OPEN NFS<2 03000005/0000000000000410 from hip-goliath CLOSE after 212 requests (181 reads, 27 writes) and 112 disk I/Os
Level 3 adds CONFIG, which reports the number of buffers used in the transaction. Single and double buffering are supported (see “Specifying the Number of Buffers”).
CONFIG 1-buf
In addition to including the bdsauto option in mount commands (see “Mounting Filesystems for BDSpro” for details), you can specify BDS services from an application. To use BDSpro, an application must open files with either the O_DIRECT or the O_BULK argument (see O_DIRECT and O_BULK descriptions in the open(2)) reference page for details). After the file is opened, these arguments may be changed with the file control function (see the FDIRECT and FBULK options of the fcntl(2)) reference page for details).
The O_DIRECT and O_BULK arguments have significant differences:
An open using O_BULK turns on write buffering, which may not be desirable for some applications (see “Using Write Buffering”). If BDS is not available, NFS is used for the transaction.
An open using O_DIRECT does not turn on write buffering. However, the open fails if BDS is not available.
Whenever direct I/O and bulk I/O are used in the same application, bulk I/O takes precedence. If write buffering is in effect and an application must be certain that data is written to disk, the application should use the fsync system call (see the fsync(2) reference page for details). BDS performance is significantly reduced when you use fsync regularly.
BDSpro recognizes the XFS fcntl command listed below (see the fcntl(2) reference page):
F_DIOINFO F_ALLOCSP F_FREESP F_RESVSP F_UNRESVSP F_ALLOCSP64 F_FREESP64 F_RESVSP64 F_UNRESVSP64 F_FSSETXATTR F_FSGETXATTR F_FSGETXATTRA |
BDSpro offers two fcntl commands that may be called by other applications to change file properties: F_GETBDSATTR and F_SETBDSATTR. Both commands take a pointer to a struct BDSATTR as an argument. Use F_GETBDSATTR first to obtain current state information, make the changes that are needed, then use F_SETBDSATTR to record the changes. All changes that you make affect all users of the file on the client until the last close of the file.
To use BDSpro from an application, include <fs/nfs/bds.h> in the application. The following excerpt is from a bds.h file:
 | Note: Do not change BDSATTR_PRIORITY. |
#define BDSATTR_BUFSIZE_DONTCARE (0xffffffffffffffffLL)
#define BDSATTR_PRIORITY_DONTCARE (0)
#define BDSATTRF_BUFFERS_SINGLE 0x01
#define BDSATTRF_WRITEBEHIND 0x08
typedef struct bdsattr {
uint64 bdsattr_flags;
uint64 bdsattr_bufsize;
u_char bdsattr_priority;
} bdsattr_t;
|