Chapter 14. FFIO Layer Reference

This chapter provides details about each of the following FFIO layers. An asterisk (*) indicates that the layer is available on IRIX systems:

Layer

Definition

blankx or blx

Blank compression/expansion layer

bmx or tape

UNICOS online tape handling

bufa *

Library-managed asynchronous buffering

c205

CDC CYBER 205 record formats

cache*

cache layer

cachea *

cachea layer

cdc

CDC 60-bit NOS/SCOPE file formats

cos *

COS blocking

er90

ER90 handling

event *

I/O monitoring (not available on CRAY T3E systems)

f77 *

UNIX record blocking

fd*

File descriptor

global*

Cache distribution layer

ibm

IBM file formats

mr

Memory-resident file handlers

nosve

CDC NOS/VE file formats

null *

The null layer

sds

SDS resident file handlers (not available on CRAY T3E systems)

syscall *

System call I/O

system *

Generic system layer

text *

Newline separated record formats

user* and site *

Writable layer

vms*

VAX/VMS file formats

Characteristics of Layers

In the descriptions of the layers that follow, the data manipulation tables use the following categories of characteristics:

Characteristic

Description

Granularity

Indicates the smallest amount of data that the layer can handle. For example, layers can read and write a single bit; other layers, such as the syscall layer, can process only 8-bit bytes. Still others, such as some CDC formats, process data in units of 6-bit characters in which any operation that is not a multiple of 6 bits results in an error.

Data model

Indicates the data model. Three main data models are discussed in this section. The first type is the record model, which has data with record boundaries, and may have an end-of-file (EOF).

The second type is stream (a stream of bits). None of these support the EOF.

The third type is the filter, which does not have a data model of its own, but derives it from the lower-level layers. Filters usually perform a data transformation (such as blank compression or expansion).

Truncate on write

Indicates whether the layer forces an implied EOD on every write operation (EOD implies truncation).

Implementation strategy

Describes the internal routines that are used to implement the layer.

The X-record type referred to under implementation strategy refers to a record type in which the length of the record is prepended and appended to the record. For f77 files, the record length is contained in 4 bytes at the beginning and the end of a record. The v type of NOS/VE and the w type of CYBER 205/ETA also prepend and append the length of the record to the record.

In the descriptions of the layers, the supported operations tables use the following categories:

Operation

Lists the operations that apply to that particular layer. The following is a list of supported operations:


ffopen                ffclose
ffread                ffflush
ffreada               ffweof
ffreadc               ffweod
ffwrite               ffseek
ffwritea              ffpos
ffwritec              ffbksp


Support

Uses three potential values: Yes, No, or Passed through. “Passed through” indicates that the layer does not directly support the operation, but relies on the lower-level layers to support it.

Used

Lists two values: Yes or No. “Yes” indicates that the operation is required of the next lower-level layer. “No” indicates that the operation is never required of the lower-level layer. Some operations are not directly required, but are passed through to the lower-layer if requested of this layer. These are noted in the comments.

Comments

Describes the function or support of the layer's function.

On many layers, you can also specify the numeric parameters by using a keyword. This functionality is available if your application is linked with Cray Research's CrayLibs 3.0 or later release. See the INTRO_FFIO(3f) man page for more details about FFIO layers.

When using direct access files on IRIX systems the user must assign the file to either the system or the global layer for code that works with more than one processor. The default layer for direct access on IRIX systems is the cache layer and it does not have the coherency to handle multiple processes doing I/O to the same file.

Individual Layers

The remaining sections in this chapter describe the individual FFIO layers in more detail.

The blankx Expansion/compression Layer (Not Available on IRIX systems)

The blankx or blx layer performs blank compression and expansion on a stream of 8-bit characters. The syntax for this layer is as follows:

blankx[.
type]:[num1]:[
num2]
blx[.
type]:[num1]:[
num2]

The keyword specification for this layer is as follows:

blankx.[
type][.blxchr=
num1][.blnk=
num2]
blx.[
type][.blxchr=
num1][.blnk=
num2]

The type field can have one of the following three values:

Value

Definition

cos

COS-style blank compression . (blxchr= 27 or 0x1D)

ctss

CTSS-style blank compression. (blxchr= 48 or 0x30)

c205

CYBER 205-style blank compression. (blxchr= 48 or 0x30)

The num1 field contains the decimal value of the ASCII character used as the escape code to control the blank compression.

The num2 field contains the decimal value of the ASCII character that is the object of the compression. This is usually the ASCII blank (0x20).

Table 14-1. Data manipulation: blankx layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Filter. Takes characteristics of lower-level layer but does some data transformation.

No

blx specific


Table 14-2. Supported operations: blankx layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Passed through

Only in lower-level layer

Yes

Only if explicitly requested

ffweod

Passed through

Only in lower-level layer

Yes

Only if explicitly requested

ffseek

No

Only seek (fd,0,0) for rewind

Yes

Only on rewind

ffpos

Yes

NA

NA

ffbksp

Passed through

Only in lower-level layer

Yes

Only if explicitly requested


The bmx/tape Layer (Deferred Implementation on IRIX systems)

The bmx or tape layer handles the interface to online magnetic tapes. The bmx layer uses the tape list I/O interface on Cray Research systems.

For information about the tmf layer on IRIX systems, see the IRIX TMF User's Guide.

A magnetic tape does not use control words to delimit records and files; however, control words are part of the physical representation of the data on the medium. On a magnetic tape, each tape block is considered a record.

The following is the syntax for this layer:

bmx:[
num1]:[num2]
tape:[num1]:[
num2]

The keyword specification is as follows:

bmx[.bufsize=
num1][.num_buffers=num2]
tape[.bufsize=num1][
.num_buffers=num2]

The num1 argument specifies the size in 512-word blocks for each buffer. The num2 argument specifies the number of buffers to use.

The bmx layer may be used with ER90 files that have been mounted in blocked mode. The ER90 device places restrictions on the amount of data that can be written to a tape block; see the Tape Subsystem User's Guide, for details.

Table 14-3 describes the EOF and EOD behavior of the bmx layer.

Table 14-3. -T specified on tpmnt

Type of tapes

EOF/EOD

No

Yes

Labeled

EOF

Never returned

At user tape marks

EOD

At end-of-file

At label/end-of-file

Unlabeled

EOF

Never returned

At user tape marks

EOD

At double tape mark

Never returned

The EOF label is always considered an EOD. For unlabeled tapes without the -T option specified, nothing can be considered an EOF. The double tape mark shows the EOD. For unlabeled tapes specified with -T, nothing can be considered an EOD and every tape mark is returned as an EOF.

No optional fields are permitted.

Table 14-4. Data manipulation: bmx/tape layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record with multiple EOF if users specify with tpmnt -T

Yes

bmx specific


Table 14-5. Supported operations: bmx/tape layer

Operation

Supported

Comments

ffopen

Yes

ffread

Yes

ffreada

Yes

Always synchronous

ffreadc

Yes

ffwrite

Yes

ffwritea

Yes

Always synchronous

ffwritec

Yes

ffclose

Yes

ffflush

Yes

ffweof

Yes

Writes tape mark if allowed

ffweod

Yes

ffseek

No

seek (fd,0,0) only (equal to rewind)

ffpos

Yes

ffbksp

Yes

Lower-level layers are not allowed. Exact implementation depends on operating system and hardware platform.

The bufa Layer

The bufa layer provides library-managed asynchronous buffering. This can reduce the number of low-level I/O requests for some files. The syntax is as follows:

bufa:[num1]:[
num2]

The keyword syntax is as follows:

bufa[.bufsize=
num1][.num_buffers=num2]

The num1 argument specifies the size, in 4096-byte blocks, of each buffer. The default buffer size depends on the device where your file is located. The maximum allowed value for num1 on IRIX systems is 32,767. The maximum allowed value on UNICOS and UNICOS/mk systems 1,073,741,823. You may not, however, be able to use a value this large because this much memory may not be available.

The num2 argument specifies the number of buffers. The default is 2.

Table 14-6. Data manipulation: bufa layer

Granularity

Data model

Truncate on write

1 bit (UNICOS and UNICOS/mk

Stream

No

8 bits (IRIX systems

Stream

No


Table 14-7. Supported operations: bufa layer

Supported operations

Required of next lower level?

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

Yes

ffweof

Passed through

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

Only if supported by the underlying layer

Yes

Only if explicitly requested

ffpos

Yes

Yes

Only if explicitly requested

ffbksp

No

No


The CYBER 205/ETA (c205) Blocking Layer (Not Available on IRIX systems)

The c205 layer performs blocking and deblocking of the native type for the CDC CYBER 205 or ETA computer systems. The general format of the specification follows:

c205.w:[recsize]:[
bufsize]

The keyword specification follows:

c205.w[.bufsize=
num2]

The w is CYBER 205 W-type records and must be specified. The recsize field should not be specified because it is reserved for future use as a maximum record size. The bufsize refers to the working buffer size for the layer and should be specified as a nonnegative decimal number (in bytes).

To achieve maximum performance, ensure that the working buffer size is large enough to completely hold any records that are written, plus the control words. Control words consist of 8 bytes per record. If a record plus control words is written larger than the buffer, the layer must perform some inefficient operations to do the write. If the buffer is large enough, these operations are avoided.

On reads, the buffer size is not as important, although larger sizes usually perform better.

If the next lower-level layer is magnetic tape, this layer does not support I/O.

Table 14-8. Data manipulation: c205 layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record

Yes. CDC end-of-group delimiter (EOG) maps to EOF, and CDC end-of-file (EOF) maps to EOD.

x records


Table 14-9. Supported operations: c205 layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No-op

No

ffweof

Yes

Mapped to end-of-group

No

ffweod

Yes

Mapped to end-of-file

Yes

ffseek

Yes

seek(fd,0,0) only (equals rewind)

Yes

Requires that the underlying interface be a stream

ffpos

Yes

NA

ffbksp

No

No


The cache Layer

The cache layer allows recently accessed parts of a file to be cached either in main memory or in a secondary data segment (SDS). This can significantly reduce the number of low-level I/O requests for some files that are accessed randomly. This layer also offers efficient sequential access when a buffered, unblocked file is needed. The syntax is as follows:

cache[
.type]:[num1]:[
num2][num3]

The following is the keyword specification:

cache[
.type][.page_size=num1][
.num_pages=num2
[.bypass_size=
num3]]

The type argument can be either mem or sds (.sds is not allowed on IRIX systems or on CRAY T3E systems). mem directs that cache pages reside in main memory; sds directs that the pages reside in secondary data segments (SDS). num1 specifies the size, in 4096-byte blocks, of each cache page buffer. The default is 8. The maximum allowed value for num1 on IRIX systems is 32,767. The maximum allowed value on UNICOS and UNICOS/mk systems 1,073,741,823. You may not, however, be able to use a value this large because this much memory may not be available.

num2 specifies the number of cache pages. The default is 4. num3 is the size in 4096-byte blocks at which the cache layer attempts to bypass cache layer buffering. If a user's I/O request is larger than num3, the request might not be copied to a cache page. The default size for num3 on IRIX systems is num3= num1. On UNICOS and UNICOS/mk systems, the default is num3= num1num2.

When a cache page must be preempted to allocate a page to the currently accessed part of a file, the least recently accessed page is chosen for preemption. Every access stores a time stamp with the accessed page so that the least recently accessed page can be found at any time.

Table 14-10. Data manipulation: cache layer

Granularity

Data model

Truncate on write

1 bit (UNICOS and UNICOS/mk systems)

Stream (mimics UNICOS system calls)

No

8 bit (IRIX systems)

Stream

No

512 words (cache.sds)

Stream

No


Table 14-11. Supported operations: cache layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

No

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

No

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

No

No

ffweod

Yes

Yes

ffseek

Yes

Yes

Requires underlying interface to be a stream

ffpos

Yes

NA

ffbksp

No

NA


The cachea Layer

The cachea layer allows recently accessed parts of a file to be cached either in main memory or in a secondary data segment (SDS). This can significantly reduce the number of low-level I/O requests for some files that are accessed randomly.

This layer can provide high write performance by asynchronously writing out selective cache pages. It can also provide high read performance by detecting sequential read access, both forward and backward. When sequential access is detected and when read-ahead is chosen, file page reads are anticipated and issued asynchronously in the direction of file access. The syntax is as follows:

cachea[type]:[
num1]:[num2]:[
num3]:[num4]

The keyword syntax is as follows:

cachea[
type][.page_size=num1][
.num_pages=num2] [.max_lead=
num3][.shared_cache=num4]

type

Directs that cache pages reside in main memory ( mem) or SDS (sds). SDS is available only on UNICOS systems.

num1

Specifies the size, in 4096-byte blocks, of each cache page buffer. Default is 8. The maximum allowed value for num1 on IRIX systems is 32,767. The maximum allowed value on UNICOS and UNICOS/mk systems 1,073,741,823. You may not, however, be able to use a value this large because this much memory may not be available.

num2

Specifies the number of cache pages to be used. Default is 4.

num3

Specifies the number of cache pages to asynchronously read ahead when sequential read access patterns are detected. Default is 0.

num4

Specifies a cache number in the range of 1 to 15. Cache number 0 is a cache which is private to the current FFIO layer. Any cache number larger than 0 is shared with any other file using a cachea layer with the same number.

Multiple cachea layers in a chain may not contain the same nonzero cache number.

On IRIX systems, stacked shared cachea layers are not supported. On UNICOS and UNICOS/mk systems, stacked shared layers are supported, but in multitasked programs, different files must not mix the order of the shared caches.

The following examples demonstrate this functionality:

  • The following specifications cannot both be used by a multitasked program:

    assign -F cachea::::1,cachea::::2 u:1
    assign -F cachea::::2,cachea::::1 u:2

  • The following specifications can both be used by a multitasked program on UNICOS systems:

    assign -F cachea::::1,cachea::::2 u:1
    assign -F cachea::::2,cachea::::1 u:2

Table 14-12. Data manipulation: cachea layer

Granularity

Data model

Truncate on write

1 bit (UNICOS and UNICOS/mk systems

Stream (mimics UNICOS system calls)

No

8 bit (IRIX systems

Stream (mimics UNICOS system calls)

No


Table 14-13. Supported operations: cachea layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

No

ffreada

Yes

Yes

ffreadc

Yes

No

ffwrite

Yes

No

ffwritea

Yes

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

No

No

ffweod

Yes

Yes

ffseek

Yes

Yes

Requires that the underlying interface be a stream

ffpos

Yes

NA

ffbksp

No

NA


The cdc Layer (Not Available on IRIX systems)

The cdc layer handles record blocking for four common record types from the 60-bit CYBER 6000 and 7000 series computer systems, which run the CDC 60-bit NOS, NOS/VE, or SCOPE operating system. The general format of the specification follows:

cdc[.recfmt].[
tpfmt]

There is no alternate keyword specification for this layer.

The supported recfmt values are as follows:

Values

Definition

iw

I-type blocks, W-type records

cw

C-type blocks, W-type records

cs

C-type blocks, S-type records

cz

C-type blocks, Z-type records

The tpfmt field can have one of the following three values that indicate the presence of block trailers and other low-level characteristics.

Field

Definition

disk

Disk type structure, for use with station transfers of CYBER data

i

NOS internal tape format

si

System internal or SCOPE internal tape format


Note: The i and si fields require a lower-level layer that handles records. A stream model in the lower-level layers does not work.

The disk field requires a lower layer that handles records when endfile makes exist prior to the end of data.


Table 14-14. Data manipulation: cdc layer

Granularity

Data model

Truncate on write

Implementation strategy

6 bits for cz records, 1 bit for iw records, and 60 bits for cs and cw records.

Record

Yes

cdc specific


Table 14-15. Supported operations: cdc layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No-op

No

ffweof

Yes

No

ffweod

Yes

Yes

ffseek

Yes

seek(fd,0,0) only (equals rewind)

Yes

seek(fd,0,0) only

ffpos

Yes

NA

ffbksp

No

No


The cos Blocking Layer

The cos layer performs COS blocking and deblocking on a stream of data. The general format of the cos specification follows:

cos:[.type][.
num1]

The format of the keyword specification follows:

cos[
.type][.bufsize=
num1]

The num1 argument specifies the working buffer size in 4096-byte blocks.

If not specified, the default buffer size is the larger of the following: the large I/O size (UNICOS and UNICOS/mk systems only); the preferred I/O block size (see the stat(2) man page for details), or 48 blocks. See the INTRO_FFIO(3f) man page for more details.

When writing, full buffers are written in full record mode, specifically so that the magnetic tape bmx layer can be used on the system side to read and write COS transparent tapes. To choose the block size of the tape, select the buffer size.

Reads are always performed in partial read mode; therefore, you do not have to know the block size of a tape to read it (if the tape block size is larger than the buffer, partial mode reads ensure that no parts of the tape blocks are skipped).

Table 14-16. Data manipulation: cos layer

Granularity

Data model

Truncate on write

Implementation strategy

1 bit

Records with multi-EOF capability

Yes

cos specific


Table 14-17. Supported operations: cos layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No-op

Yes

ffweof

Yes

No

ffweod

Yes

Yes

Truncation occurs only on close

ffseek

Yes

Minimal support (see following note)

Yes

ffpos

Yes

NA

ffbksp

Yes

No records

No


Note: seek operations are supported only to allow for rewind (seek(fd,0,0)), seek-to-end ( seek(fd,0,2)) and a GETPOS(3f) or SETPOS(3f) operation, where the position must be on a record boundary.


GETPOS(3f) and SETPOS(3f) are not available on IRIX systems .

The er90 Layer (Available Only on UNICOS Systems)

The er90 layer is not supported on IRIX systems or on CRAY T3E systems. It is available only on UNICOS systems.

The er90 layer handles the interface to the ER90 files. No arguments are accepted.

Table 14-18. Data manipulation: er90 layer

Granularity

Data model

Truncate on write

8 bits

Stream

Yes


Table 14-19. Supported operations: er90 layer

Operation

Supported

Comments

ffopen

Yes

ffread

Yes

ffreada

Yes

ffreadc

No

ffwrite

Yes

ffwritea

Yes

ffwritec

Yes

ffclose

Yes

ffflush

Yes

ffweof

No

ffweod

Yes

ffseek

Yes

ffseek(fd,0,0) only (equals rewind)

ffbksp

No

Lower-level layers are not allowed.

The event Layer

The event layer monitors I/O activity (on a per-file basis) which occurs between two I/O layers. It generates statistics as an ASCII log file and reports information such as the number of times an event was called, the event wait time, the number of bytes requested, and so on. You can request the following types of statistics:

  • A list of all event types

  • Event types that occur at least once

  • A single line summary of activities that shows information such as amount of data transferred and the data transfer rate.

Statistics are reported to stderr by default. The FF_IO_LOGFILE environment variable can be used to name a file to which statistics are written by the event layer. The default action is to overwrite the existing statistics file if it exists. You can append reports to the existing file by specifying a plus sign (+ ) before the file name, as in this example:

setenv FF_IO_LOGFILE +saveIO

This layer report counts for read, reada , write, and writea. These counts represent the number of calls made to an FFIO layer entry point. In some cases, the system layer may actually use a different I/O system call, or multiple system calls. For example, the reada system call does not exist on IRIX systems, and the system layer reada entry point will use aio_read() .

On IRIX systems, amention of the lock layer may be included during report generation even though that layer may not have been specified by the user.

On CRAY T3E systems, if more than one PE is using the event layer, and you set the FF_IO_LOGFILE environment variable, you must use the plus sign ( +) before the file name to prevent PE a from overwriting the information written by PE b. Using the plus sign also means that the information will be appended to an existing file.

On CRAY T3E systems, you can also use the FF_IO_LOGFILEPE environment variable to name a file to which statistics are written. The file name will be x.n, where x is the name specified by the environment variable and n is the number of the PE which wrote the file. The default action is to overwrite the existing file. To append information to an existing file, specify a plus sign ( +) before the file name.

The event layer is enabled by default and is included in the executable file; you do not have to relink to study the I/O performance of your program. To obtain event statistics, rerun your program with the event layer specified on the assign command, as in this example:

assign -F bufa, event, cachea, event, system

The syntax for the event layer is as follows:

event[.type]

There is no alternate keyword specification for this layer.

The type argument selects the level of performance information to be written to the ASCII log file; it can have one of the following values:

Value

Definition

nostat

No statistical information is reported.

summary

Event types that occur at least once are reported.

brief

A one line summary for layer activities is reported.

The f77 Layer

The f77 layer handles blocking and deblocking of the f77 record type, which is common to most UNIX Fortran implementations. The syntax for this layer is as follows:

f77[.type]:[
num1]:[num2]

The following is the syntax of the keyword specification:

f77[.type][.recsize=
num1][.bufsize=num2]

The type argument specifies the record type and can take one of the following two values:

Value

Definition

nonvax

Control words in a format common to large machines such as the MC68000; default.

vax

VAX format (byte-swapped) control words.

The num1 field refers to the maximum record size. The num2 field refers to the working buffer size.

To achieve maximum performance, ensure that the working buffer size is large enough to hold any records that are written plus the control words (control words consist of 8 bytes per record). If a record plus control words are larger than the buffer, the layer must perform some inefficient operations to do the write. If the buffer is large enough, these operations can be avoided.

On reads, the buffer size is not as important, although larger sizes will usually perform better.

If the next lower-level layer is magnetic tape, this layer does not support I/O.

Table 14-20. Data manipulation: f77 layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record

Yes

x records


Table 14-21. Supported operations: f77 layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Passed through

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

ffseek(fd,0,0) equals rewind; ffseek(fd,0,2) seeks to end

Yes

ffpos

Yes

NA

ffbksp

Yes

Only in lower-level layer

No


The fd Layer

The fd layer allows connection of a FFIO file to a system file descriptor. You must specify the fd layer, as follows:

fd:[num1]

The keyword specification is as follows:

fd[
.file_descriptor=num1]

The num1 argument must be a system file descriptor for an open file. The ffopen or ffopens request opens a FFIO file descriptor that is connected to the specified file descriptor. The file connection does not affect the file whose name is passed to ffopen.

All other properties of this layer are the same as the system layer. See “The system Layer”, for details.

The global Layer

The global layer is a caching layer that distributes data across all multiple SHMEM or MPI processes. Open and close operations require participation by all processes which access the file; all other operations are independently performed by one or more processes.

The following is the syntax for the global layer:

global[. type]:[
num1]:[num2]

The following is the syntax for the keyword specification:

global[. type][.page_size=
num1][.num_pages=num2]

The type argument can be privpos (default), in which is the file position is private to a process or globpos (deferred implementation), in which the file position is global to all processes.

The num1 argument specifies the size in 4096-byte blocks of each cache page. num2 specifies the number of cache pages to be used on each process. If there are n processes, then n num2 cache pages are used.

num2 buffer pages are allocated on every process which shares access to a global file. File pages are direct-mapped onto processes such that page n of the file will always be cached on process (n mod NPES), where NPES is the total number of processes sharing access to the global file. Once the process is identified where caching of the file page will occur, a least-recently-used method is used to assign the file page to a cache page within the caching process.

Table 14-22. Data manipulation: global layer

Granularity

Data model

Truncate on write

8 bits

Stream

No


Table 14-23. Supported operations: global layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

No

ffreada

Yes

Always synchronous

Yes

ffreadc

Yes

No

ffwrite

Yes

No

ffwritea

Yes

Always synchronous

Yes

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

No

No

ffweod

Yes

Yes

ffseek

Yes

Yes

Requires underlying interface to be a stream

ffpos

Yes

NA

ffbksp

No

NA


The ibm Layer (Deferred Implementation on IRIX systems)

The ibm layer handles record blocking for seven common record types on IBM operating systems. The general format of the specification follows:

ibm.[type]:[
num1]:[num2]

The keyword specification follows:

ibm[.
type][.recsize=num1][
.mbs=num2]

The supported type values are as follows:

Value

Definition

u

IBM undefined record type

f

IBM fixed-length records

fb

IBM fixed-length blocked records

v

IBM variable-length records

vb

IBM variable-length blocked records

vbs

IBM variable-length blocked spanned records

The f format is fixed-length record format. For fixed-length records, num1 is the fixed record length (in bytes) for each logical record. Exactly one record is placed in each block.

The fb format records are the same as f format records except that you can place more than one record in each block. num1 is the length of each logical record. num2 must be an exact multiple of num1.

The v format records are variable-length records. recsize is the maximum number of bytes in a logical record. num2 must exceed num1 by at least 8 bytes. Exactly one logical record is placed in each block.

The vb format records are variable-length blocked records. This means that you can place more than one logical record in a block. num1 and num2 are the same as with v format.

The vbs format records have no limit on record size. Records are broken into segments, which are placed into one or more blocks. num1 should not be specified. When reading, num2 must be at least large enough to accommodate the largest physical block expected to be encountered.

The num1 field is the maximum record size that may be read or written. The vbs record type ignores it.

The num2 (maximum block size) field is the maximum block size that the layer uses on reads or writes.

Table 14-24. Values for maximum record size on ibm layer

Field

Minimum

Maximum

Default

Comments

u

1

32,760

32,760

f

1

32,760

None

Required

fb

1

32,760

None

Required

v

5

32,756

32,752

Default is num2-8 if not specified

vb

5

32,756

32,752

Default is num2-8 if not specified

vbs

1

None

None

No maximum record size

Table 14-25. Values for maximum block size in ibm layer

Field

Minimum

Maximum

Default

Comments

u

1

32,760

32,760

Should be equal to num1

f

1

32,760

num1

Must be equal to num1

fb

1

32,760

num1

Must be multiple of num1

v

9

32,760

32,760

Must be >= num1 + 8

vb

9

32,760

32,760

Must be >= num1 + 8

vbs

9

32,760

32,760


Table 14-26. Data manipulation: ibm layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record

No for f and fb records. Yes for v, vb, and vbs records.

f records for f and fb. v records for u, v, vb, and vbs .


Table 14-27. Supported operations: ibm layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Passed through

Yes

ffweod

Yes

Yes

ffseek

Yes

seek(fd, 0, 0) only (equals rewind)

Yes

seek(fd,0,0) only

ffpos

Yes

NA

ffbksp

No

No


The mr Layer (Deferred Implementation on IRIX systems)

The memory-resident ( mr) layer lets users declare that a file should reside in memory. The mr layer tries to allocate a buffer large enough to hold the entire file.

The options are as follows:

mr[.type
[.subtype]]:
num1:num2:num3

The keyword specification is as follows:

mr[.
type[.subtype]][
.start_size=num1][.max_size=
num2]
[.inc_size=
num3]

The type field specifies whether the file in memory is intended to be saved or is considered a scratch file. This argument accepts the following values:

Value

Definition

save

Loads (reads) as much of the file as possible into memory when the file is opened (if it exists). If the data in memory is changed, the file data is written back to the next lower layer at close time. The save option also modifies the behavior of overflow processing. save is the default.

scr

Does not try to load at open and discards data on close (scratch file). The scr option also modifies the behavior of overflow processing.

The subtype field specifies the action to take when the data can no longer fit in the allowable memory space. It accepts the following values:

Value

Definition

ovfl

Excess data that does not fit in the specified medium is written to the next lower layer. ovfl is the default value.

novfl

When the memory limit is reached, any further operations that try to increase the size of the file fail.

The num1, num2, and num3 fields are nonnegative integer values that state the number of 4096-byte blocks to use in the following circumstances:

Field

Definition

num1

When the file is opened, this number of blocks is allocated for the file. Default: 0.

num2

This is the limit on the total size of the memory space allowed for the file in this layer. Attempted growth beyond this limit causes either overflow or operation failure, depending on the overflow option specified. Default: 246-1

num3

This is the minimum number of blocks that are allocated whenever more memory space is required to accommodate file growth. Default: 256 for SDS files and 32 for memory resident files.

The num1 and num3 fields represent best-effort values. They are intended for tuning purposes and usually do not cause failure if they are not satisfied precisely as specified (for example, if the available memory space is only 100 blocks and the chosen num3 value is 200 blocks, growth is allowed to use the 100 available blocks rather than failing to grow, because the full 200 blocks requested for the increment are unavailable).

When using the mr layer, large memory-resident files may reduce I/O performance for sites that provide memory scheduling that favors small processes over large processes. Check with your system administrator if I/O performance is diminished.


Caution:: Use of the default value for the max parameter can cause program failure if the file grows and exhausts the entire amount of memory available to the process. If the file size might become quite large, always provide a limit.


Memory allocation is done by using the malloc (3c) and realloc(3c) library routines. The file space in memory is always allocated contiguously.

When allocating new chunks of memory space, the num3 argument is used in conjunction with realloc as a minimum first try for reallocation.

Table 14-28. Data manipulation: mr layer

Primary function

Granularity

Data model

Truncate on write

Avoid I/O to the extent possible, by holding the file in memory.

1 bit

Stream (mimics UNICOS system calls)

No


Table 14-29. Supported operations: mr layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

Sometimes delayed until overflow

ffread

Yes

Yes

Only on open

ffreada

Yes

No

ffreadc

Yes

No

ffwrite

Yes

Yes

Only on close, overflow

ffwritea

Yes

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No-op

No

ffweof

No

No representation

No

No representation

ffweod

Yes

Yes

ffseek

Yes

Full support (absolute, relative, and from end)

Yes

Used in open and close processing

ffpos

Yes

NA

ffbksp

No

No records

No


The nosve Layer (Not Available on IRIX systems)

The nosve layer handles record blocking for five common record types on CDC NOS/VE operating systems.

The general format of the specifications is as follows:

nosve[
.type]:[num1]:[
num2]

The format of the keyword specifications is as follows:

nosve[
.type][.recsize=
num1][.mbs=num2]

The supported type fields follow:

Field

Definition

v

NOS/VE format record

f

ANSI F fixed-length records

s

ANSI S format (segmented) records

d

ANSI D format (variable-length) records

u

NOS/VE undefined record

The num1 field is the maximum record size that can be read or written. The s and v record types ignore it.

Table 14-30. Values for maximum record size

recfmt

Minimum

Maximum

Default

Comments

v

1

No maximum

None

f

1

65,536

None

Required

s

1

No maximum

None

d

1

9,995

4,123

u

1

32,760

32,760


Table 14-31. Values for maximum block size

recfmt

Minimum

Maximum

Default

Comments

v

32

Memory size

32,768

Working buffer size

f

1

65,536

num1

s

6

32,767

4,128

d

5

32,767

4,128

u

1

32,760

32,760

For the nosve.v format, the working buffer size can affect performance. If the buffer size is at least as large as the largest record that will be written, the system overhead is minimized. For the nosve.u record format, num1 and num2 are the same thing. For nosve.f and nosve.d records, the maximum block size must be at least as large as the maximum record size. You can place more than one record in a block.

For nosve.s records, one or more segments are placed in each block (a record is composed of one or more segments).

Table 14-32. Data manipulation: nosve layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record

No for f records. Yes for u, s, d, and v records.

f records for f. v records for u, s, and d. x records for v.


Table 14-33. Supported operations: nosve layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Passed through

Yes for s records; passed through for others

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

ffseek(fd,0,0) only (equals rewind)

Yes

Extensively for nosve.v

ffpos

Yes

NA

ffbksp

No

No


The null layer

The null layer is a syntactic convenience for users; it has no effect. This layer is commonly used to simplify the writing of a shell script when a shell variable is used to specify a FFIO layer specification. For example, the following is a line from a shell script with a tape file using the assign command and overlying blocking is expected on the tape (as specified by BLKTYP):

assign -F $BLKTYP,bmx fort.1

If BLKTYP is undefined, the illegal specification list ,bmx results. The existence of the null layer lets the programmer set BLKTYP to null as a default, and simplify the script, as in the following:

assign -F null,bmx fort.1

This is identical to the following command:

assign -F bmx fort.1

The sds Layer (Available Only on UNICOS Systems)

The sds layer is not available on CRAY T3E systems or on IRIX systems.

The sds layer lets users declare that a file should reside on SDS. The specification for this layer follows:

sds[
.type:[subtype]]:[
num1]:[num2]:[
num3]

The keyword specification is as follows:

sds[.
type[.subtype]][
.start_size=num1][.max_size=
num2]
[.inc_size=
num3]

The type field specifies whether the file to reside in SDS is intended to be saved. This field can have the following values:

Value

Definition

save

Loads (reads) as much of the file as possible into SDS as soon as the file is opened (if it exists). If the data in SDS is changed, the SDS data is written back to the next lower layer at close time. The save option also modifies the behavior of overflow. save is the default.

scr

Does not attempt to load at open and discards data on close (scratch file). The scr option also modifies the behavior of overflow processing.

The subtype field specifies the action to take when the data can no longer fit in the allowable SDS space. It can have the following values:

Value

Definition

ovfl

Excess data that does not fit in the specified medium is written to the next lower layer. This is the default.

novfl

When the SDS limit is reached, any further operations that try to increase the size of the file fails.

The num1, num2, and num3 fields are nonnegative integer values that state the number of 4096-byte blocks to use in the following circumstances.

Field

Definition

num1

When the file is opened, this number of blocks is allocated for the file.

num2

This is the limit on the total size of the SDS space allowed for the file in this layer. Attempted growth beyond this limit causes either overflow or operation failure, depending on the overflow option specified.

num3

This is the minimum number of blocks that are allocated each time more SDS space is required to accommodate file growth.

The num1 and num3 fields are used for tuning purposes and usually do not fail if they are not used precisely as specified. For example, if the available SDS space is only 100 blocks, and the chosen increase (num3) value is 200 blocks, growth is allowed to use the 100 available blocks instead of failing to grow (because the full 200 blocks requested for the increment are unavailable). Similarly, the num3 value of 200 implies allocation in minimum size chunks of 200 blocks. If 200 blocks of contiguous space is unavailable, the allocation is satisfied with whatever space is available.

The specification for sds is equivalent to the following specification:

sds.save.ovfl:0:35184372088832:256

Overflow is provided when the requested data cannot completely reside in SDS. This can occur either because the SDS space requested from the system is not available or because the num2 (maximum size) argument was specified.

When overflow occurs, a message prints to standard error stating the file name and the overflow size. The overflow I/O to the next lower layer depends on the type argument. If save is specified, the sds layer assumes that the part of the file that resides in SDS must eventually be written to the lower-level layer (usually disk).

The overflowed data is written to the lower-level layer at a position in the file that corresponds to the position it will occupy after the SDS-resident data is flushed. Space is reserved at overflow time in the file to accommodate the SDS resident part of the file.

If the scr option is selected, the SDS resident part of the file is considered disposable. Space for it is not reserved in the file on the lower-level layer. The overflow operations behave as though the first overflowed bit in the file is bit 0 of the lower-level layer, as in the following example:

# requests a max of 1 512-word block of SDS
assign -F sds.save.ovfl:0:1 fort.1

Assume that the file does not initially exist. The initial SDS size is 0 blocks, and the size is allowed to grow to a maximum of 1 block. If a single write of 513 words was done to this file, the first 512 words are written to SDS. The remaining word is written to file fort.1 at word position 512.

Words 0 through 511 are not written until the sds layer is closed and the SDS data is flushed to the lower-level layer. Immediately after the write completes, SDS contains 512 words, and fort.1 consists of 513 words. Only the last word contains valid data until the file is closed.

If the assign command is of the following form, it is assumed that the entire file is disposable if 513 words are written to the file:

# requests a max of 1 512-word block of SDS
assign -F sds.scr.ovfl:0:1 fort.1

It is not necessary to reserve space in fort.1 for the SDS data. When the 513 words are written to the file, the first 512 words are written to SDS. The 513th word is written to word 0 of fort.1. After the completion of the write, fort.1 consists of 1 word. The fort.1 file is deleted when the file is closed.

SDS allocation is done through the sdsalloc (3f) library routine. The file space in SDS is allocated (as far as possible) in a contiguous manner, but if contiguous space is not found, any available fragments are used before overflow is forced on any file.

When allocating new chunks of SDS space, the num3 argument is used as a minimum first try for allocation.

Table 14-34. Data manipulation: sds layer

Primary function

Granularity

Data model

Truncate on write

The sds layer lets the users obtain the fastest possible I/O rates through the SDS hot path.

1 bit

Stream (mimics UNICOS system calls)

No


Table 14-35. Supported operations: sds layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

Sometimes delayed until overflow

ffread

Yes

Yes

Only on open

ffreada

Yes

No

ffreadc

Yes

No

ffwrite

Yes

Yes

Only on close, overflow

ffwritea

Yes

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

Flushes only internal buffer not SDS

No

ffweof

No

No

No representation

ffweod

Yes

No representation

Yes

ffseek

Yes

Full support (absolute, relative, and from end)

Yes

Used in open and close processing

ffpos

Yes

NA

ffbksp

No

No records

No

No records


The syscall Layer

The syscall layer directly maps each request to an appropriate system call. The layer does not accept any options on UNICOS or UNICOS/mk systems. On IRIX systems, it has one optional parameter, as follows:

syscall[
.cboption]

The cboption argument can have one of the following values:

aiocb

The syscall layer will be notified, via a signal, when the asynchronous I/O is completed.

noaiocb

The syscall layer will poll the completion status word to determine asynchronous I/O completion. This is the default value.

Table 14-36. Data manipulation: syscall layer

Granularity

Data model

Truncate on write

8 bits (1 byte)

Stream (UNICOS system calls)

No

Table 14-37. Supported operations: syscall layer

Operation

Supported

Comments

ffopen

Yes

open

ffread

Yes

read

ffreada

Yes

reada( aio.read on IRIX systems

ffreadc

Yes

read plus code

ffwrite

Yes

write

ffwritea

Yes

writea (aio.write on IRIX systems

ffwritec

Yes

write plus code

ffclose

Yes

close

ffflush

Yes

None

ffweof

No

None

ffweod

Yes

trunc(2)

ffseek

Yes

lseek(2)

ffpos

Yes

ffbksp

No


Lower-level layers are not allowed.

The system Layer

The system layer is implicitly appended to all specification lists, if not explicitly added by the user (unless the syscall, tape, er90, or fd layer is specified). It maps requests to appropriate system calls.

If the file that is opened is a tape file, the system layer becomes the tape layer.

For a description of options, see the syscall layer. Lower-level layers are not allowed.

The text Layer

The text layer performs text blocking by terminating each record with a newline character. It can also recognize and represent the EOF mark. The text layer is used with character files and does not work with binary data. The general specification follows:

text[.
type]:[num1]:[
num2]

The keyword specification follows:

text[.
type][.newline=num1][
.bufsize=num2]

The type field can have one of the following three values:

Value

Definition

nl

Newline-separated records.

eof

Newline-separated records with a special string such as ~e. More than one EOF in a file is allowed.

c205

CYBER 205-style text file (on the CYBER 205, these are called R-type records).

The num1 field is the decimal value of a single character that represents the newline character. The default value is 10 (octal 012, ASCII line feed).

The num2 field specifies the working buffer size (in decimal bytes). If any lower-level layers are record oriented, this is also the block size.

Table 14-38. Data manipulation: text layer

Granularity

Data model

Truncate on write

8 bits

Record.

No


Table 14-39. Supported operations: text layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Passed through

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

Yes

ffpos

Yes

No

ffbksp

No

No


The user and site Layers

The user and site layers let users and site administrators build layers that meet specific needs. The syntax follows:

user[num1]:[
num2]
site:[
num1]:[num2]

The open processing passes the num1 and num2 arguments to the layer and are interpreted by the layers.

See “Creating a user Layer,” Chapter 15, “Creating a user Layer ” for an example of how to create an FFIO layer.

The vms Layer

The vms layer handles record blocking for three common record types on VAX/VMS operating systems. The general format of the specification follows.

vms.[type.subtype]:[
num1]:[num2]

The following is the alternate keyword specification for this layer:

vms.[type.subtype][
.recsize=num1][.mbs=
num2]

The following type values are supported:

Value

Definition

f

VAX/VMS fixed-length records

v

VAX/VMS variable-length records

s

VAX/VMS variable-length segmented records

In addition to the record type, you must specify a record subtype, which has one of the following four values:

Value

Definition

bb

Format used for binary blocked transfers

disk

Same as binary blocked

tr

Transparent format, for files transferred as a bit stream to and from the VAX/VMS system

tape

VAX/VMS labeled tape

The num1 field is the maximum record size that may be read or written. It is ignored by the s record type.

Table 14-40. Values for record size: vms layer

Field

Minimum

Maximum

Default

Comments

v.bb

1

32,767

32,767

v.tape

1

9995

2043

v.tr

1

32,767

2044

s.bb

1

None

None

No maximum record size

s.tape

1

None

None

No maximum record size

s.tr

1

None

None

No maximum record size

The num2 field is the maximum segment or block size that is allowed on input and is produced on output. For vms.f.tr and vms.f.bb, num2 should be equal to the record size (num1). Because vms.f.tape places one or more records in each block, vms.f.tape num2 must be greater than or equal to num1.

Table 14-41. Values for maximum block size: vms layer

Field

Minimum

Maximum

Default

Comments

v.bb

1

32,767

32,767

v.tape

6

32,767

2,048

v.tr

3

32,767

32,767

N/A

s.bb

5

32,767

2,046

s.tape

7

32,767

2,048

s.tr

5

32,767

2,046

N/A

For vms.v.bb and vms.v.disk records, num2 is a limit on the maximum record size. For vms.v.tape records, it is the maximum size of a block on tape; more specifically, it is the maximum size of a record that will be written to the next lower layer. If that layer is tape, num2 is the tape block size. If it is cos, it will be a COS record that represents a tape block. One or more records are placed in each block.

For segmented records, num2 is a limit on the block size that will be produced. No limit on record size exists. For vms.s.tr and vms.s.bb, the block size is an upper limit on the size of a segment. For vms.s.tape , one or more segments are placed in a tape block. It functions as an upper limit on the size of a segment and a preferred tape block size.

Table 14-42. Data manipulation: vms layer

Granularity

Data model

Truncate on write

Implementation strategy

8 bits

Record

No for f records. Yes for v and s records.

f records for f formats. v records for v formats.


Table 14-43. Supported operations: vms layer

Supported operations

Required of next lower level?

Operation

Supported

Comments

Used

Comments

ffopen

Yes

Yes

ffread

Yes

Yes

ffreada

Yes

Always synchronous

No

ffreadc

Yes

No

ffwrite

Yes

Yes

ffwritea

Yes

Always synchronous

No

ffwritec

Yes

No

ffclose

Yes

Yes

ffflush

Yes

No

ffweof

Yes and passed through

Yes for s records; passed through for others

Yes

Only if explicitly requested

ffweod

Yes

Yes

ffseek

Yes

seek(fd,0,0) only (equals rewind)

Yes

seek(fd,0,0) only

ffpos

Yes

NA

ffbksp

No

No