This chapter describes the design and architecture of IRIS HIPPI. The functionality described in this chapter is directly available to programs that use the character device API. For programs using the IRIX sockets API (for example, to use the services of the IP protocol stack), this functionality is hidden by the network stack.
This section provides an overview of IRIS HIPPI.
The Silicon Graphics implementations of HIPPI provide the services and conform to the protocols described in the HIPPI standards for HIPPI-PH, HIPPI–Serial (when appropriate), HIPPI–FP, the host portion of HIPPI-SC, and HIPPI-LE. Also see sections, “How HIPPI Protocol Items Are Handled With the HIPPI-PH Access Method”, and “How HIPPI Protocol Items Are Handled With the HIPPI-FP Access Method”.
IRIS HIPPI supports transmission and reception as separate channels. Because of this design, it is possible for a system to have applications that only receive, applications that only send, and/or applications that do both. In addition, each channel can be accessed with a different access method. (The access methods are described in section, “The Device File and Access Methods”.)
 | Note: Due to the nature of HIPPI-Serial, simultaneous use of the channels to different endpoints requires the presence of a HIPPI switch between each source/destination pair. For each HIPPI-PH connection over HIPPI-Serial, the switch demultiplexes the backflowing control signals (for example, READYs). |
For transmission, IRIS HIPPI supports four different functionality scenarios, summarized in Table 1-1 and described in the text below the table. Each functionality scenario is a combination of one connection control method (the rows of Table 1-1) and one packet control method (the columns of Table 1-1). The bullets describe these two control methods:
Connection control: A connection can be single-packet or many-packet. A single-packet connection is when one packet is sent and then the HIPPI subsystem automatically closes the connection. A many-packet connection is a connection that is kept open for as long as the application wants. In the latter case, the application must indicate when it wants the connection closed.
Packet control: A packet can be single-write or multiple-write. A single-write packet is a HIPPI packet that is created by the HIPPI subsystem from an application's single write() call. A multiple-write packet is created from two or more write() calls. In the latter case, the application must indicate the start of each packet.
Single-packet connection, single-write packet:
The application does one write(), which causes the connection to be opened.
One packet is sent. It consists of the data in the write() call.
The HIPPI subsystem automatically closes the connection.
Many-packet connection, single-write packets:
The application asks for a long-term connection.
The application does the first write() call and the connection is opened.
One packet is sent. It consists of the data in the write() call.
The application does any number of write() calls.
One packet is created and sent from each write().
The connection remains open until the application closes it.
Single-packet connection, multiple-write packet:
The application indicates the length of a packet.
The application does the first write() call, which causes the connection to be opened.
The application continues to do write() calls.
A packet is sent. Data from the write() calls, up to the specified packet length, are sent as one packet. The packet length can be indeterminate.
The HIPPI subsystem closes the connection.
Many-packet connection, multiple-write packets:
The application asks for a long-term connection.
The application indicates the length of a packet. This length may be indeterminate or a specified bytecount.
The application does the first write() call and the connection is opened.
The application does any number of write() calls.
A packet is sent. Data from the write() calls, up to the specified packet length, are sent as one packet. The packet length can be indeterminate.
The application may indicate the end of the packet at any time.
The application can send any number of packets by indicating a length for each new packet.
The connection remains open until the application closes it.
The IRIS HIPPI implementation consists of 4 main components:
the device file (/dev/hippi#) and access methods for controlling the HIPPI subsystem, illustrated in Figure 1-1
transmission-related information objects (FDOs) for each open device file descriptor
reception-related information objects (ULPOs) for different upper-layer protocol applications (ULPs)
the application programming interface (API)
Each of these components is described in a section below.
A portion of the IRIS HIPPI subsystem, within the UNIX kernel, maintains transmission information objects, referred to as file descriptor objects (FDOs). An FDO is maintained for each open file descriptor. The HIPPI subsystem uses this information for generating HIPPI packets on transmission. Applications change the FDO values through the IRIS HIPPI API. The values are persistent, so they can be used on sequentially sent packets, without resetting.
the ULP's identification number (that is, the ULP-id)
access mode (read-only, write-only, read and write)
the I–field used when establishing a connection
the size of the first burst for each packet
the setting for the B-bit in the FP header (used by HIPPI-FP only)
the setting for the P-bit of the FP header (used by HIPPI-FP only)
the size of the D1 area (used by HIPPI-FP only)
A portion of the IRIS HIPPI subsystem, within the UNIX kernel, maintains reception information objects, called upper-layer protocol objects (ULPOs). A ULPO is maintained for each ULP-id. Each ULPO consists of a set of information that the HIPPI subsystem uses when receiving HIPPI packets. Each application must have a ULPO associated with (bound to) it. With HIPPI-FP, the information in one ULPO can be shared among a group of applications. Applications use the HIPPI API (ioctl calls) to change their ULPO values.
| Note: HIPPI-LE (over which TCP/IP runs) is an example of a ULP.[1] ULPs that customers may develop include IPI-3 and “raw” protocols. |
Each ULPO maintains the following information:
the ULP's identification number (that is, the ULP-id), used only by HIPPI-FP
the number of applications using this ULP-id, used only by HIPPI-FP
the received bytecount for the packet currently being read
The IRIS HIPPI product includes 2 application programming interfaces (APIs): a character device API composed of ioctl() calls and an IRIX sockets API that uses the standard BSD-socket calls.
The character device API allows customer-developed applications to change the information in their ULPO and FDO and to control the HIPPI subsystem. This API is through a UNIX “character special” device file. By invoking different IRIS HIPPI API commands, customer-developed programs define their access method, data flow (packet) control, connection control, and HIPPI protocol processing.
Details on the character device API are provided in Chapter 2, “Programming Notes for Character Device API.” Details for the socket API are provided in Chapter 3, “Programming Notes for Sockets-based API.”
The IRIS HIPPI implementation provides the /dev/hippi# device file for accessing and controlling the HIPPI subsystem. Two different access methods (described below) are provided. Both use the /dev/hippi# device file. The access method is defined when the device file is bound.
IRIS HIPPI offers two mutually exclusive methods for accessing the HIPPI subsystem: HIPPI-FP and HIPPI-PH. The HIPPI-FP access method requires the use of FP headers and provides automatic processing of that header. The HIPPI-PH method does not require use of the FP header, thus allowing an application to bypass the HIPPI-FP layer.
Besides the difference (discussed above) in the point of access, the main differences between the two access methods revolve around coexistence with other ULPs (including HIPPI-LE and the TCP/IP stack), as listed below:
For HIPPI-FP, received packets are demultiplexed using ULP-ids. For HIPPI-PH, all packets are placed on a single input queue.
For HIPPI-FP, access to the HIPPI subsystem is shared among two or more ULPs (of which the IP protocol stack is one). With HIPPI-FP, the HIPPI device is blocked when a ULP is accessing the device, thus giving each ULP exclusive access for the duration of its access. With the HIPPI-PH access method, no blocking occurs because only one ULP is allowed to be bound to the HIPPI device at any time.
Further details are provided in sections, “The HIPPI-PH Access Method”, and “The HIPPI-FP Access Method”.
The Silicon Graphics IRIS HIPPI Product (HIPPI board, device driver, interface, and firmware) has been designed to meet the ANSI X3T11 Standards Committee's standards for the High-Performance Parallel Interface.[2] The HIPPI board design includes the following implementation details that are either not mentioned in the ANSI documentation for the HIPPI standard, or are considered by the design team to be ambiguously defined in the standards documentation:
Once a receiving channel has been created and configured to accept connections, all incoming connection requests are accepted by the HIPPI board. The HIPPI subsystem does not wait for upper-layer input. (That is, the HIPPI board does not generate the service primitive PH_RING.Indicate and does not allow the application to respond with a PH_ANSWER.Request for each connection). The upper layers may discard data that has been received from undesirable connections.
Each instance of a ULPO must be assigned a ULP-id that is unique within the ULPOs for a specific HIPPI board. A valid ULP-id is a number between 0 and 255 decimal (inclusive); 4 is reserved for and used by the IRIX module implementing 8802.2 Link Encapsulation (HIPPI-LE); 7 is reserved for IPI-3 implementations.
IRIS HIPPI includes a ULP module (HIPPI-LE) for servicing the IP stack. This module can coexist with customer-developed applications; however, IP network traffic has requirements that, in some situations, conflict with full usage of some features of the HIPPI subsystem. These requirements are:
IP applications need short but frequent access to the HIPPI subsystem.
IP requires bi-directional flow of data (for example, data packets in one direction require acknowledgment packets in the other direction).
Customer–developed applications can coexist well with IP applications if they follow these guideline:
They must not create (open) long-term connections. Since only one connection at a time can use a HIPPI channel, if any one ULP monopolizes the channel, the IP performance degrades significantly.
They must limit their write()s to small-sizes (for example, 128kbytes).
They must read() their input queue in a timely fashion so that host buffers are always available to retrieve packets from the board. Arriving packets are enqueued in the IRIS HIPPI board's memory and are transferred from the board into host memory on a first-arrived-first-transferred basis. As long as a packet in the queue is not read, all subsequently arriving packets queue up behind it. IP performance degrades when packet latencies become excessive. Additionally, the onboard memory can fill up and arriving packets can be dropped.
The physical connections for both channels must attach to the same node. For example, with a copper-based product, both cables must be attached to the same switch or HIPPI node.
| Note: In addition to degrading the performance of local IP applications, failure to follow these guidelines may cause wide performance variations or hangs at intermediate switches or at the endpoints with which the station is exchanging data. |
The HIPPI-PH access method controls the HIPPI protocol stack at the HIPPI-PH signal layer. This access method is independent of the physical layer, so it works for both copper-based and fiber optic-based hardware. With this access method, the HIPPI-FP protocol is bypassed; the IRIS HIPPI subsystem does no checking for or processing of HIPPI-FP protocol items. Accessing the HIPPI subsystem in this manner is well-suited for applications requiring full or almost full use of the HIPPI device, and situations where (for other reasons) the application does not wish to use the HIPPI-FP protocol.
HIPPI–PH supports the following functions for reception:
Accepts all incoming HIPPI packets. Does not reject any packet, and does not demultiplex using the ULP-id.
Enqueues the entire packet on the input queue for retrieval by the application. Does not interpret anything in the packet (not an FP header or D1 data).
The HIPPI subsystem maintains a received bytecount value (offset) that can be used by applications to identify packet boundaries
HIPPI–PH supports the following functions for transmission:
Provides two choices for setting up the HIPPI connection: a single-packet connection (where the HIPPI subsystem creates and tears down a connection for each packet), and a long-term connection (where the HIPPI subsystem keeps the connection up across one, many, or all packets). In long-term connections, the application controls the timing of the disconnect.
Provides two methods for creating packets: “multiple-write” packets (where the PACKET signal is asserted across multiple write() calls) and single-write packets (where the PACKET signal is deasserted when the data from one write() has been transmitted). The maximum bytecount for any write() is 16 megabytes, so a single-write packet cannot be larger than 16 megabytes.
Allows an application to send an “infinite” sized packet.
Allows an application to specify that the first burst of any packet be a short burst.
Allows an application to terminate a multiple-write packet before its bytecount is transmitted.
Allows an application to use the HIPPI subsystem as a “raw” data pipeline. For example, the FP header is not required and the I-field can be set to any value.
HIPPI-PH supports the four different functionality scenarios for transmission, summarized in Table 1-1.
Applications using the HIPPI-PH access method can open the device for transmit only (illustrated in Figure 1-2), receive only (also illustrated in Figure 1-2), or for both (illustrated in Figure 1-3). Once the device is open and bound, any of the functionality scenarios in Table 1-1 can be used.
Figure 1-2. Block Diagram for Two Applications Using HIPPI-PH: One Receive-Only and One Transmit-Only
 |
There are certain constraints associated with using the HIPPI-PH access method, as listed below. However, with HIPPI-PH overall, there are fewer constraints on what the application can do with the interface than with HIPPI-FP.
If more than one application operates a channel (transmit or receive) of the HIPPI board, an arbitration and synchronization mechanism between the applications must be developed to prevent race conditions.
The HIPPI network interface cannot be ifconfig'ed up, which means that the TCP/IP protocol stack cannot use the HIPPI board.
For receiving, if a HIPPI-FP header exists in the packet, it is not interpreted (and not demultiplexed) by the HIPPI subsystem.
All read()s and write()s must specify buffers that are 8-byte word–aligned. This is because direct memory access (DMA) occurs directly to/from user application space and the HIPPI device only handles word–aligned DMAs.
The data lengths for all read()s and write()s must be multiples of 8 bytes up to a maximum of 16 megabytes.
When a device is opened for writing and bound with the HIPPI-PH access method, the HIPPI subsystem transmits only data that the application passes to it. No additional data, encapsulation, or HIPPI protocols are added by the HIPPI subsystem. The only information used from the application's FDO is the I-field and the short burst setting.
The application can define the first burst as short within each packet.
When a device is opened for reading and bound with the HIPPI-PH access method, the HIPPI subsystem receives all inbound data; all packets are enqueued on the reading queue. The HIPPI subsystem does not attempt to interpret an FP header; therefore, if an FP/D1 header exists, these are passed to the application as part of the data stream. No demultiplexing is performed on the ULP-id. No special handling features are available.
The application can retrieve a packet bytecount (offset) value that simplifies identification of packet boundaries.
This section describes how the HIPPI-PH access method handles the HIPPI I-field and the HIPPI Framing Protocol.
The FDO maintains a value for the I-field. The HIPPI subsystem uses the value each time it sets up a connection. Applications use an ioctl() call to set the I-field value to a new one whenever desired. The HIPPI subsystem does not interpret or alter the I-field in any way during transmission.
The HIPPI subsystem does not interpret or alter the I-field in any way for reception.
The HIPPI-PH access method does not generate an FP header or D1 data area for packets on transmission. An application may utilize the HIPPI Framing Protocol by generating its own FP header/D1 data and transmitting these just as it does all other packet data (with write calls), as illustrated in Figure 1-4. An application may invoke an ioctl() call to define the bytecount for the first (short) burst; the data for that first burst is taken from the first write() call, as illustrated in Figure 1-5.
On incoming packets, HIPPI-PH does not check for the presence of an FP header nor does it interpret the FP header if one exists. If an FP header/D1 area are present in a packet, the HIPPI subsystem treats that packet just as it does any other packet (enqueuing the contents of the packet on the receive queue without any interpretation, separation, or special processing).
The HIPPI-FP access method controls the HIPPI protocol stack at the HIPPI-FP layer and provides for sharing of the HIPPI receive and/or transmit channels among applications, as illustrated in Figure 1-6. Up to 32 different applications (open file descriptors) can use the IRIS HIPPI subsystem simultaneously in any combination of transmitting-only, receiving-only, and transmitting-and-receiving.
HIPPI–FP supports the following functions for reception:
Up to 32 different applications can simultaneously have open file descriptors for receiving HIPPI packets.
Up to 8 different customer-developed ULPOs can be active simultaneously. Each application must bind to one ULPO. HIPPI-IPI is an example of a ULPO. (The HIPPI-LE module that is part of the HIPPI product does not count as one of these ULPOs.)
The HIPPI subsystem demultiplexes incoming packets using the ULP-ids from the active ULPOs. It discards packets that do not match any of the ULP-ids in active ULPOs.
For HIPPI-FP packets, the HIPPI subsystem verifies the presence of a valid FP header and discards packets that do not have a valid FP header.
For HIPPI-FP packets, the HIPPI subsystem separates the FP header and D1 data from the D2 data so that the application's first read() retrieves the FP header and D1 data, while its subsequent read()s retrieve the D2 data.
The HIPPI subsystem maintains a packet offset value (bytecount) that can be used by character device applications to identify packet boundaries.
User-layer applications can use the IRIX socket interface to access the services of network–layer stacks: the default stack is the Internet Protocol (IP) suite. The HIPPI product ships with a socket-based driver, if_hip, that supports IP–over–HIPPI (using HIPPI–LE). Customer-developed character device programs can coexist with socket-based programs. (This coexistence is not available in “mixed” configurations, where one of the HIPPI channels is being used by HIPPI-FP and the other by HIPPI-PH.)
Note: The IP software requires equal-access to its lower layer services. If applications sharing the HIPPI subsystem with IP do not meet this requirement, the performance of IP is seriously compromised. Special “auto-bind” device files can be set up with more general permissions in order to allow user access to specific ULPOs.
HIPPI–FP supports the following functions for transmission:
Up to 32 different applications can simultaneously have open file descriptors for transmitting HIPPI packets.
Up to 8 different customer-developed FDOs can be active simultaneously. Each application must bind to one FDO. HIPPI-IPI is an example of an FDO. (The HIPPI-LE module that is part of the HIPPI product does not count as one of these FDOs.)
The HIPPI subsystem creates an FP header for each packet.
Allows an application to specify that the first burst of any packet contains only the FP header and, optionally, a D1 area. The word count of this first burst can be short (1 to 255 words) or standard (256 words). The B bit in the FP header is automatically set.
Allows an application to specify the presence of D1 data and the size of the D1 area. The P bit in the FP header is automatically set.
HIPPI-FP supports the four different functionality scenarios for transmission, summarized in Table 1-1.
With HIPPI-FP, there are certain constraints, as listed below:
For transmitting, a HIPPI-FP header is attached to every outgoing packet. If the application does not specify an FP header, the HIPPI subsystem uses a default one (illustrated in Figure 1-7).
For receiving, each incoming packet must have a valid HIPPI-FP header. The HIPPI subsystem demultiplexes incoming packets based upon the ULP identifier in the FP header.
Incoming connection requests cannot be selectively rejected by the ULPO or application; each incoming connection request results in acceptance of the packet. However, the HIPPI subsystem discards packets with ULP-ids that do not match any of those that are currently bound. All applications that have opened a HIPPI file descriptor for receiving (reading) and have bound to a ULPO will receive all incoming packets destined to the bound ULP-id.
All read()s and write()s must specify buffers that are 8-byte word–aligned. This is because direct memory access (DMA) occurs directly to or from user application space and the HIPPI device only handles word–aligned DMAs.
The data lengths for all read()s and write()s must be multiples of 8 bytes up to a maximum of 16 megabytes.
The destination endpoint is specified by an ioctl() call that sets the I-field for all subsequent packets (until the value is changed). The I-field can be changed at any time, and the ioctl() call is efficient enough that there is no problem with setting the I-field just before each write() call for a series of single-write packets.
HIPPI FP headers are automatically generated on output. The default FP header (illustrated in Figure 1-7) has the ULP identifier (specified at bind time), does not have any D1 area, and the P and B bits are off. If an application defines a size for the D1 area or specifies a first burst containing only FP header and D1 area, this information is included in the header. The D2_Size field in the FP header is filled with the proper value.
When a device is opened for reading and bound to a ULPO with the HIPPI-FP access method, the associated application is able to retrieve all HIPPI packets that arrive with the bound ULP-id. The demultiplexing on ULP-ids is done on the HIPPI board so that DMA can occur directly to user-space. If multiple applications share a ULPO, demultiplexing the packets must be handled by an application-level program.
HIPPI-FP separates the FP header and D1_Data_Area from the D2 area. The application's first read() call returns the FPheader/D1data exactly. (The return value tells how large these areas are). Subsequent read() calls return the D2 area until the D2 data is completely read. By monitoring the HIPPI subsystem's packet offset value, the application can tell when the next read() is going to return the FPheader/D1area for a new packet.
It is possible to receive packets that are very large because reception can be broken up into multiple read()s. This also helps provide for some scattering of data, but small read()s are inefficient.
An application can retrieve a packet bytecount (offset) value that simplifies identification of packet boundaries.
This section describes how the HIPPI-FP access method handles the HIPPI I-field and the HIPPI Framing Protocol.
Each FDO contains a value for the I-field that the HIPPI subsystem includes each time it sets up a connection. Applications use an ioctl() call to set the value to a new one whenever desired. The HIPPI subsystem does not interpret nor alter the I-field in any way during transmission.
The HIPPI subsystem does not interpret nor alter the I-field in any way during reception.
When accessed with the HIPPI-FP access method, the HIPPI subsystem uses the HIPPI Framing Protocol on all connections for receiving and transmitting, as explained below.
The HIPPI subsystem creates an FP header for each packet that it transmits. The default value for the generated FP header is as follows:
D2_Size (that is, bits 63:32 of the FP header) is set to the size of the write() call for a single-write packet or, for a multiple-write packet, the bytecount indicated by the ioctl() call that starts the packet.
ULP-id (that is, bits 31:24) is set to the ULP-id that was provided by the application when it bound to the ULPO.
Control (P and B) and reserved bits are off (that is, bits 23:11 are set to zero).
D1_Area_Size and D2_Offset are set to zero.
To include D1 data in a packet, an application specifies the size of the D1 area, using an ioctl() call. This action sets the D1_Area_Size in the FDO and causes the P bit in the FP header to be set ON. The application then does its write pointing to contiguous D1 and D2 data, or for a multiple-write packet, it does the first write pointing to D1 data or contiguous D1 and D2 data.
To place only the FP header and D1 data (optional) in the first burst and to set the B-bit, an application invokes an ioctl() call, specifying the size of the first burst. If the specified size is less than 256 words, the IRIS HIPPI subsystem handles the burst as a short burst. The HIPPI subsystem creates a first burst of the indicated size (short or standard length) that contains the following data:
The required, and automatically generated, FP header (8 bytes).
An optional number of D1 data bytes, up to a maximum of 1016.
The D2 size for the FP header is calculated by the IRIS HIPPI subsystem, as described immediately below.
For a single-write packet, the D2 data size is the size of the write() call minus the D1_Area_Size, as shown in Figure 1-8.
For any multiple-write packet, the D2 data size is as specified by an ioctl() command, as shown in Figure 1-9 (illustrating contiguous D1 and D2 data) and Figure 1-10 (illustrating FP header and D1 data separated into the first burst).
With each reception of a packet, the HIPPI subsystem interprets the FP header information, as required by the standard. The application's first read() retrieves the FPheader/D1data; subsequent read() calls retrieve D2 data. It is the responsibility of the reading application(s) to keep track of which read()s retrieve which kinds of data. The HIPPI subsystem demultiplexes incoming packets, using the ULP-id field. Incoming packets for unrecognized ULP-ids are discarded by the HIPPI subsystem.
HIPPI-PH and HIPPI-FP access methods can be used simultaneously so that they share one IRIS HIPPI board. There are some restrictions for this configuration:
Each HIPPI-channel (receive or transmit) must be used by either HIPPI-PH or HIPPI-FP, but not both. For example, a number of applications and ULPOs can use HIPPI-FP for receiving demultiplexed data while a sending application uses HIPPI-PH, as illustrated in Figure 1-11.
The TCP/IP over HIPPI-LE protocol stack cannot be supported because it requires HIPPI-FP access for both transmit and receive.
A switch must be used between each the endpoints of each connection.
