Chapter 13. FFIO Layer Reference

This chapter provides details about each of the following FFIO layers:

Layer

Definition

bufa

Library-managed asynchronous buffering

cache

cache layer

cachea

cachea layer

cos

COS blocking

event

I/O monitoring

f77

UNIX record blocking

fd

File descriptor

global

Cache distribution layer

null

The null layer

syscall

System call I/O

system

Generic system layer

text

Newline separated record formats

user and site

Writable layer

vms

VAX/VMS file formats

Characteristics of Layers

In the descriptions of the layers that follow, the data manipulation tables use the following categories of characteristics:

Characteristic

Description

Granularity

Indicates the smallest amount of data that the layer can handle. For example, layers can read and write a single bit; other layers, such as the syscall layer, can process only 8-bit bytes. Still others, such as some CDC formats, process data in units of 6-bit characters in which any operation that is not a multiple of 6 bits results in an error.

Data model

Indicates the data model. Three main data models are discussed in this section. The first type is the record model, which has data with record boundaries, and may have an end-of-file (EOF).

The second type is stream (a stream of bits). None of these support the EOF.

The third type is the filter, which does not have a data model of its own, but derives it from the lower-level layers. Filters usually perform a data transformation (such as blank compression or expansion).

Truncate on write

Indicates whether the layer forces an implied EOD on every write operation (EOD implies truncation).

Implementation strategy

Describes the internal routines that are used to implement the layer.

The X-record type referred to under implementation strategy refers to a record type in which the length of the record is prepended and appended to the record. For f77 files, the record length is contained in 4 bytes at the beginning and the end of a record. The v type of NOS/VE and the w type of CYBER 205/ETA also prepend and append the length of the record to the record.

In the descriptions of the layers, the supported operations tables use the following categories:

Operation

Lists the operations that apply to that particular layer. The following is a list of supported operations:

ffopen                ffclose
ffread                ffflush
ffreada               ffweof
ffreadc               ffweod 
ffwrite               ffseek 
ffwritea              ffpos
ffwritec              ffbksp 

Support

Uses three potential values: Yes, No, or Passed through. “Passed through” indicates that the layer does not directly support the operation, but relies on the lower-level layers to support it.

Used

Lists two values: Yes or No. “Yes” indicates that the operation is required of the next lower-level layer. “No” indicates that the operation is never required of the lower-level layer. Some operations are not directly required, but are passed through to the lower-layer if requested of this layer. These are noted in the comments.

Comments

Describes the function or support of the layer's function.

On many layers, you can also specify the numeric parameters by using a keyword. See the INTRO_FFIO(3f) man page for more details about FFIO layers.

When using direct access files, you must assign the file to either the system or the global layer for code that works with more than one processor. The default layer for direct access is the cache layer and it does not have the coherency to handle multiple processes doing I/O to the same file.

Individual Layers

The remaining sections in this chapter describe the individual FFIO layers in more detail.

The bufa Layer

The bufa layer provides library-managed asynchronous buffering. This can reduce the number of low-level I/O requests for some files. The syntax is as follows:

bufa:[num1]:[num2]

The keyword syntax is as follows:

bufa[.bufsize=num1][.num_buffers=num2]

The num1 argument specifies the size, in 4096-byte blocks, of each buffer. The default buffer size depends on the device where your file is located. The maximum allowed value for num1 on IRIX systems is 32,767.

The num2 argument specifies the number of buffers. The default is 2.

Table 13-1. Data manipulation: bufa layer

Granularity

Data model

Truncate on write

8 bits

Stream

No


Table 13-2. Supported operations: bufa layer

Supported operations

Required of next lower level?

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

Yes

ffweof

Passed through

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

Only if supported by the underlying layer

Yes

Only if explicitly requested

ffpos

Yes

Yes

Only if explicitly requested

ffbksp

No

No


The cache Layer

The cache layer allows recently accessed parts of a file to be cached either in main memory. This can significantly reduce the number of low-level I/O requests for some files that are accessed randomly. This layer also offers efficient sequential access when a buffered, unblocked file is needed. The syntax is as follows:

cache[.type]:[num1]:[num2][num3]

The following is the keyword specification:

cache[.type][.page_size=num1][.num_pages=num2
[.bypass_size=num3]]

The type argument must be mem; this directs that cache pages reside in main memory. num1 specifies the size, in 4096-byte blocks, of each cache page buffer. The default is 8. The maximum allowed value for num1 is 32,767.

num2 specifies the number of cache pages. The default is 4. num3 is the size in 4096-byte blocks at which the cache layer attempts to bypass cache layer buffering. If a user's I/O request is larger than num3, the request might not be copied to a cache page. The default size for num3 is num3=num1.

When a cache page must be preempted to allocate a page to the currently accessed part of a file, the least recently accessed page is chosen for preemption. Every access stores a time stamp with the accessed page so that the least recently accessed page can be found at any time.

Table 13-3. Data manipulation: cache layer

Granularity

Data model

Truncate on write

8 bit

Stream

No

512 words (cache.sds)

Stream

No


Table 13-4. Supported operations: cache layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

No

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

No

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

No

No

ffweod

Yes

Yes

ffseek

Yes

Yes

Requires underlying interface to be a stream

ffpos

Yes

NA

ffbksp

No

NA


The cachea Layer

The cachea layer allows recently accessed parts of a file to be cached either in main memory. This can significantly reduce the number of low-level I/O requests for some files that are accessed randomly.

This layer can provide high write performance by asynchronously writing out selective cache pages. It can also provide high read performance by detecting sequential read access, both forward and backward. When sequential access is detected and when read-ahead is chosen, file page reads are anticipated and issued asynchronously in the direction of file access. The syntax is as follows:

cachea[mem]:[num1]:[num2]:[num3]:[num4]

The keyword syntax is as follows:

cachea[mem][.page_size=num1][.num_pages=num2] [.max_lead=num3][.shared_cache=num4]

mem

Directs that cache pages reside in main memory.

num1

Specifies the size, in 4096-byte blocks, of each cache page buffer. Default is 8. The maximum allowed value for num1 is 32,767.

num2

Specifies the number of cache pages to be used. Default is 4.

num3

Specifies the number of cache pages to asynchronously read ahead when sequential read access patterns are detected. Default is 0.

num4

Specifies a cache number in the range of 1 to 15. Cache number 0 is a cache which is private to the current FFIO layer. Any cache number larger than 0 is shared with any other file using a cachea layer with the same number.

Multiple cachea layers in a chain may not contain the same nonzero cache number.

Stacked shared cachea layers are not supported.

The following examples demonstrate this functionality:

  • The following specifications cannot both be used by a multitasked program:

    assign -F cachea::::1,cachea::::2 u:1
    assign -F cachea::::2,cachea::::1 u:2

Table 13-5. Data manipulation: cachea layer

Granularity

Data model

Truncate on write

8 bit

Stream

No


Table 13-6. Supported operations: cachea layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

No

ffreada

Yes

Yes

ffreadc

Yes

No

ffwrite

Yes

No

ffwritea

Yes

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

No

No

ffweod

Yes

Yes

ffseek

Yes

Yes

Requires that the underlying interface be a stream

ffpos

Yes

NA

ffbksp

No

NA


The cos Blocking Layer

The cos layer performs COS blocking and deblocking on a stream of data. The general format of the cos specification follows:

cos:[.type][.num1]

The format of the keyword specification follows:

cos[.type][.bufsize=num1]

The num1 argument specifies the working buffer size in 4096-byte blocks.

If not specified, the default buffer size is the larger of the following: the preferred I/O block size (see the stat(2) man page for details), or 8 4096-byte blocks. See the INTRO_FFIO(3f) man page for more details.

Reads are always performed in partial read mode; therefore, you do not have to know the block size of a tape to read it (if the tape block size is larger than the buffer, partial mode reads ensure that no parts of the tape blocks are skipped).

Table 13-7. Data manipulation: cos layer

Granularity

Data model

Truncate on write

Implementation strategy

1 bit

Records with multi-EOF capability

Yes

cos specific


Table 13-8. Supported operations: cos layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No-op

Yes

ffweof

Yes

No

ffweod

Yes

Yes

Truncation occurs only on close

ffseek

Yes

Minimal support (see following note)

Yes

ffpos

Yes

NA

ffbksp

Yes

No records

No


The event Layer

The event layer monitors I/O activity (on a per-file basis) which occurs between two I/O layers. It generates statistics as an ASCII log file and reports information such as the number of times an event was called, the event wait time, the number of bytes requested, and so on. You can request the following types of statistics:

  • A list of all event types

  • Event types that occur at least once

  • A single line summary of activities that shows information such as amount of data transferred and the data transfer rate.

Statistics are reported to stderr by default. The FF_IO_LOGFILE environment variable can be used to name a file to which statistics are written by the event layer. The default action is to overwrite the existing statistics file if it exists. You can append reports to the existing file by specifying a plus sign (+) before the file name, as in this example:

setenv FF_IO_LOGFILE +saveIO

This layer report counts for read, reada, write, and writea. These counts represent the number of calls made to an FFIO layer entry point. In some cases, the system layer may actually use a different I/O system call, or multiple system calls. For example, the reada system call does not exist on IRIX systems, and the system layer reada entry point will use aio_read().

Amention of the lock layer may be included during report generation even though that layer may not have been specified by the user.

The event layer is enabled by default and is included in the executable file; you do not have to relink to study the I/O performance of your program. To obtain event statistics, rerun your program with the event layer specified on the assign command, as in this example:

assign -F bufa, event, cachea, event, system

The syntax for the event layer is as follows:

event[.type]

There is no alternate keyword specification for this layer.

The type argument selects the level of performance information to be written to the ASCII log file; it can have one of the following values:

Value

Definition

nostat

No statistical information is reported.

summary

Event types that occur at least once are reported.

brief

A one line summary for layer activities is reported.

The f77 Layer

The f77 layer handles blocking and deblocking of the f77 record type, which is common to most UNIX Fortran implementations. The syntax for this layer is as follows:

f77[.type]:[num1]:[num2]

The following is the syntax of the keyword specification:

f77[.type][.recsize=num1][.bufsize=num2]

The type argument specifies the record type and can take one of the following two values:

Value

Definition

nonvax

Control words in a format common to large machines such as the MC68000; default.

vax

VAX format (byte-swapped) control words.

The num1 field refers to the maximum record size. The num2 field refers to the working buffer size.

To achieve maximum performance, ensure that the working buffer size is large enough to hold any records that are written plus the control words (control words consist of 8 bytes per record). If a record plus control words are larger than the buffer, the layer must perform some inefficient operations to do the write. If the buffer is large enough, these operations can be avoided.

On reads, the buffer size is not as important, although larger sizes will usually perform better.

If the next lower-level layer is magnetic tape, this layer does not support I/O.

Table 13-9. Data manipulation: f77 layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record

Yes

x records


Table 13-10. Supported operations: f77 layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Passed through

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

ffseek(fd,0,0) equals rewind; ffseek(fd,0,2) seeks to end

Yes

ffpos

Yes

NA

ffbksp

Yes

Only in lower-level layer

No


The fd Layer

The fd layer allows connection of a FFIO file to a system file descriptor. You must specify the fd layer, as follows:

fd:[num1]

The keyword specification is as follows:

fd[.file_descriptor=num1]

The num1 argument must be a system file descriptor for an open file. The ffopen or ffopens request opens a FFIO file descriptor that is connected to the specified file descriptor. The file connection does not affect the file whose name is passed to ffopen.

All other properties of this layer are the same as the system layer. See “The system Layer”, for details.

The global Layer

The global layer is a caching layer that distributes data across all multiple SHMEM or MPI processes. Open and close operations require participation by all processes which access the file; all other operations are independently performed by one or more processes.

The following is the syntax for the global layer:

global[. type]:[num1]:[num2]

The following is the syntax for the keyword specification:

global[. type][.page_size=num1][.num_pages=num2]

The type argument can be privpos (default), in which is the file position is private to a process or globpos (deferred implementation), in which the file position is global to all processes.

The num1 argument specifies the size in 4096-byte blocks of each cache page. num2 specifies the number of cache pages to be used on each process. If there are n processes, then n num2 cache pages are used.

num2 buffer pages are allocated on every process which shares access to a global file. File pages are direct-mapped onto processes such that page n of the file will always be cached on process (n mod NPES), where NPES is the total number of processes sharing access to the global file. Once the process is identified where caching of the file page will occur, a least-recently-used method is used to assign the file page to a cache page within the caching process.

Table 13-11. Data manipulation: global layer

Granularity

Data model

Truncate on write

8 bits

Stream

No


Table 13-12. Supported operations: global layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

No

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

No

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

No

No

ffweod

Yes

Yes

ffseek

Yes

Yes

Requires underlying interface to be a stream

ffpos

Yes

NA

ffbksp

No

NA


The null layer

The null layer is a syntactic convenience for users; it has no effect. This layer is commonly used to simplify the writing of a shell script when a shell variable is used to specify a FFIO layer specification. For example, the following is a line from a shell script with a tape file using the assign command and overlying blocking is expected on the tape (as specified by BLKTYP):

assign -F $BLKTYP,bmx fort.1

If BLKTYP is undefined, the illegal specification list ,bmx results. The existence of the null layer lets the programmer set BLKTYP to null as a default, and simplify the script, as in the following:

assign -F null,bmx fort.1

This is identical to the following command:

assign -F bmx fort.1

The syscall Layer

The syscall layer directly maps each request to an appropriate system call. It has one optional parameter, as follows:

syscall[.cboption]

The cboption argument can have one of the following values:

aiocb

The syscall layer will be notified, via a signal, when the asynchronous I/O is completed.

noaiocb

The syscall layer will poll the completion status word to determine asynchronous I/O completion. This is the default value.

Table 13-13. Data manipulation: syscall layer

Granularity

Data model

Truncate on write

8 bits (1 byte)

Stream

No

Table 13-14. Supported operations: syscall layer

Operation

Supported

Comments

ffopen

Yes

open

ffread

Yes

read

ffreada

Yes

reada(aio.read on IRIX systems

ffreadc

Yes

read plus code

ffwrite

Yes

write

ffwritea

Yes

writea (aio.write on IRIX systems

ffwritec

Yes

write plus code

ffclose

Yes

close

ffflush

Yes

None

ffweof

No

None

ffweod

Yes

trunc(2)

ffseek

Yes

lseek(2)

ffpos

Yes

ffbksp

No


Lower-level layers are not allowed.

The system Layer

The system layer is implicitly appended to all specification lists, if not explicitly added by the user (unless the syscall, or fd layer is specified). It maps requests to appropriate system calls.

If the file that is opened is a tape file, the system layer becomes the tape layer.

For a description of options, see the syscall layer. Lower-level layers are not allowed.

The text Layer

The text layer performs text blocking by terminating each record with a newline character. It can also recognize and represent the EOF mark. The text layer is used with character files and does not work with binary data. The general specification follows:

text[.type]:[num1]:[num2]

The keyword specification follows:

text[.type][.newline=num1][.bufsize=num2]

The type field can have one of the following three values:

Value

Definition

nl

Newline-separated records.

eof

Newline-separated records with a special string such as ~e. More than one EOF in a file is allowed.

c205

CYBER 205-style text file (on the CYBER 205, these are called R-type records).

The num1 field is the decimal value of a single character that represents the newline character. The default value is 10 (octal 012, ASCII line feed).

The num2 field specifies the working buffer size (in decimal bytes). If any lower-level layers are record oriented, this is also the block size.

Table 13-15. Data manipulation: text layer

Granularity

Data model

Truncate on write

8 bits

Record.

No


Table 13-16. Supported operations: text layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Passed through

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

Yes

ffpos

Yes

No

ffbksp

No

No


The user and site Layers

The user and site layers let users and site administrators build layers that meet specific needs. The syntax follows:

user[num1]:[num2]

site:[num1]:[num2]

The open processing passes the num1 and num2 arguments to the layer and are interpreted by the layers.

See “Creating a user Layer,” Chapter 14, “Creating a user Layer ” for an example of how to create an FFIO layer.

The vms Layer

The vms layer handles record blocking for three common record types on VAX/VMS operating systems. The general format of the specification follows.

vms.[type.subtype]:[num1]:[num2]

The following is the alternate keyword specification for this layer:

vms.[type.subtype][.recsize=num1][.mbs=num2]

The following type values are supported:

Value

Definition

f

VAX/VMS fixed-length records

v

VAX/VMS variable-length records

s

VAX/VMS variable-length segmented records

In addition to the record type, you must specify a record subtype, which has one of the following four values:

Value

Definition

bb

Format used for binary blocked transfers

disk

Same as binary blocked

tr

Transparent format, for files transferred as a bit stream to and from the VAX/VMS system

tape

VAX/VMS labeled tape

The num1 field is the maximum record size that may be read or written. It is ignored by the s record type.

Table 13-17. Values for record size: vms layer

Field

Minimum

Maximum

Default

Comments

v.bb

1

32,767

32,767

v.tape

1

9995

2043

v.tr

1

32,767

2044

s.bb

1

None

None

No maximum record size

s.tape

1

None

None

No maximum record size

s.tr

1

None

None

No maximum record size

The num2 field is the maximum segment or block size that is allowed on input and is produced on output. For vms.f.tr and vms.f.bb, num2 should be equal to the record size (num1). Because vms.f.tape places one or more records in each block, vms.f.tape num2 must be greater than or equal to num1.

Table 13-18. Values for maximum block size: vms layer

Field

Minimum

Maximum

Default

Comments

v.bb

1

32,767

32,767

v.tape

6

32,767

2,048

v.tr

3

32,767

32,767

N/A

s.bb

5

32,767

2,046

s.tape

7

32,767

2,048

s.tr

5

32,767

2,046

N/A

For vms.v.bb and vms.v.disk records, num2 is a limit on the maximum record size. For vms.v.tape records, it is the maximum size of a block on tape; more specifically, it is the maximum size of a record that will be written to the next lower layer. If that layer is tape, num2 is the tape block size. If it is cos, it will be a COS record that represents a tape block. One or more records are placed in each block.

For segmented records, num2 is a limit on the block size that will be produced. No limit on record size exists. For vms.s.tr and vms.s.bb, the block size is an upper limit on the size of a segment. For vms.s.tape, one or more segments are placed in a tape block. It functions as an upper limit on the size of a segment and a preferred tape block size.

Table 13-19. Data manipulation: vms layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record

No for f records. Yes for v and s records.

f records for f formats. v records for v formats.


Table 13-20. Supported operations: vms layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Yes and passed through

Yes for s records; passed through for others

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

seek(fd,0,0) only (equals rewind)

Yes

seek(fd,0,0) only

ffpos

Yes

NA

ffbksp

No

No