Chapter 1. What is HIPPI?

This chapter is an introduction to the High-Performance Parallel Interface (HIPPI) protocol, including HIPPI-PHY, HIPPI–Serial, and HIPPI–FP. The chapter provides a brief introduction to HIPPI, a description of the HIPPI protocol, some common configurations of HIPPI equipment, and how to obtain official HIPPI documentation.

Introduction to the HIPPI Protocol

This section provides a brief introduction to HIPPI.

HIPPI Terminology

HIPPI uses source to refer to the transmitting endpoint, host, network interface, or program.

It uses destination to refer to the receiving endpoint, host, network interface, or program.

A word in the HIPPI environment can be either 4 bytes (32 bits) or 8 bytes (64 bits), depending on the HIPPI implementation. In 800 megabit per second implementations, each word is 4 bytes. When not clarified, both definitions apply. For example, “A burst consists of 256 words” means that a burst can be either 1024 bytes (256 words times 4 bytes) or 2048 bytes (256 words times 8 bytes).

How HIPPI Works

HIPPI is an extremely fast, point-to-point protocol. HIPPI provides for transmission at 800 or 1600 megabits per second.[1] Before data can be sent from one HIPPI endpoint to another, there must be both a physical link and a negotiated open connection between them. For its physical layer, a HIPPI implementation can use either the HIPPI-PH or the HIPPI-Serial standard. For HIPPI-PH, each physical link is a copper cable up to 25-meters long. For HIPPI-Serial, each link consists of a fiber-optic cable that can be from 2 to 10,000 meters long (depending on the type of cable and optics).[2] For both HIPPI–PH and HIPPI–Serial, each physical link connects two HIPPI nodes. Each node can be an endpoint or an intermediate HIPPI switch, as illustrated in Figure 1-1, where an endpoint-to-endpoint configuration is illustrated as well as 2 examples of configurations that include switches.

Figure 1-1. Examples of Links Between HIPPI Nodes


In copper-based implementations, the HIPPI hardware has two 100–pin connectors for connecting copper cables, as illustrated in Figure 1-2; each cable is a single link that carries upper-layer data in one direction and control data in both directions. In HIPPI-Serial implementations, the hardware has 1 connector for a dual-fiber fiber–optic cable, as illustrated in Figure 1-2; the fiber-optic cable contains 2 physical links; each link carries upper–layer and control data in one direction. Some of the control data on each fiber relates to the connection (that is, the data stream) on the other fiber in the same cable. In fiber-based implementations, switches must demultiplex the control data that flows in the opposite direction from the data, as illustrated in Figure 1-2.

Figure 1-2. User and Control Data Carried by HIPPI-PH and HIPPI-Serial Links


The open connection is an agreement for the transfer of upper–layer data from one endpoint to another. To open a connection, the two endpoints exchange HIPPI signals (control data) according to the protocol specified in the HIPPI-PH standard. (HIPPI-Serial uses the same control signals as HIPPI-PH.) In many HIPPI implementations, each link is designed as an independent entity, so that links can each support a connection to a different host,. as illustrated by each link_2 example in Figure 1-2.


Note: With HIPPI-PH, Host A can connect directly to both Host-B and Host-E, without the use of a switch in between. With HIPPI-Serial, Host-C can connect directly to Host-D without a switch; however, for Host-C to connect to Host-D and Host-F, as illustrated, an intermediate switch (or similar device for multiplexing the control data) must be used.

Figure 1-3 illustrates a configuration of HIPPI equipment with six different physical links and nine possible endpoint-to-endpoint connections (listed below), of which three can be simultaneously active (engaged in open connections):

  • A-source transmitting (over link_1) to any one of the following:

    - A-destination (itself, over link_2)

    - B-destination (link_4)

    - C-destination (link_6)

  • B-source transmitting (over link_3) to A-destination (link_2), B-destination (itself, over link_4)), or C-destination (link_6)

  • C-source transmitting (over link_5) to A-destination (link_2), B-destination (link_4), or C-destination (itself, over link_6))

    Figure 1-3. HIPPI Links and Connections


An open connection consists of an exchange of signals (control data) between a source and a destination. (Figure 1-17 illustrates the signals.) During this exchange, the destination agrees to accept data exclusively from the source; throughout the transfer of the upper–layer data, the destination uses backflowing signals to inform the source of its ability to accept more data. Each link supports only one connection (that is, HIPPI is point–to–point). To move data in both directions between two hosts, two endpoint-to-endpoint links (or series of links) and two connections are needed between the two hosts.

Unlike Ethernet, 802.5 Token Ring, or FDDI, HIPPI does not use a shared medium. Once a connection is established, the physical link (or links) between the two HIPPI interfaces contains data packets transmitted only by the source (that is, HIPPI connections are simplex). HIPPI packets may be seen by intermediate switches but not by other host interfaces. A connection may be kept open for extended periods of time, even when there is no data moving across it, or it may be closed by either endpoint at any time; however, each endpoint may not participate in another connection until the current one has been closed.

HIPPI communication is controlled by three basic functions: connection control, packet and flow control, and routing control (pertinent only when one or more switches are involved). Each of these is discussed separately in the subsections that follow.

Connection Control

One of the first things any HIPPI endpoint does upon startup is to assert its two outgoing INTERCONNECT signals and to look for assertion of its two incoming INTERCONNECT signals. Each channel (the source and the destination) has both an incoming and an outgoing INTERCONNECT signal. When both signals on a channel are asserted, the physical link between the local system and the system at the other end is ready for use. When the other system is a switch, the exchange of INTERCONNECT signals occurs between the endpoint and the switch, not between endpoints.

Before a source (transmitting) HIPPI interface can send a packet, it must open a connection to one destination HIPPI endpoint. The source interface is always the initiator for opening the connection. To open a connection, the sender issues a connection request by asserting the REQUEST signal on the link. Each connection request includes an I-field (described in the section “The I-field”). The I-field contains (among other things) routing information, used by switches along the path to the destination.

The destination endpoint accepts a connection by asserting its CONNECT signal in response to the request. If the destination endpoint is unwilling to accept the connection or if there is a problem with the connection request (for example, bad parity on the I-field or incompatible word size), the connection request is denied (that is, acknowledged, then rejected). The transmitter must wait and try again later or forgo the communication. If the destination is unreachable (for example, a broken physical link, a powered-down or dysfunctional interface), there is no response and the source program times out.

When a switch exists between the source and destination, the source receives its connection rejections from the switch, not directly from the destination. The rejection can be caused by any of the following conditions, and it is not possible to distinguish among them (except as explained below):

  1. The destination is malfunctioning.

  2. The destination refuses to accept the requested connection.

  3. The connection request has an error.

  4. At least one of the physical links enroute to the destination is busy (currently engaged in another connection).

A feature is available that allows the source to be informed of rejections that are due to error conditions (items 1-3 above) but not to be bothered when the rejection is due to a busy link (item 4). This feature is called camp-on. By setting the camp-on bit in the I-field, the source can program the switches to hold onto the connection request until the busy link to the destination becomes available.

When the camp-on bit is set, the first switch enqueues the connection request if it finds any link along the path to the destination busy. The switch periodically checks to see if the link has become available. When the link becomes available, it sends the connection request. A switch continues to wait until it sends the REQUEST to the ultimate destination endpoint or until the source aborts the connection request. If a number of sources are all trying to send data through the same link, the camp-on feature ensures fair (first come, first served) access to the link.

Once opened, a HIPPI connection may be kept open for as long as the two endpoints maintain it. Either endpoint may terminate the connection at any time; however, the source interface is usually the one to do this.

Packet and Flow Control

Once a connection is open, one or multiple packets may be sent. The destination indicates it is ready to receive data by sending a READY signal to the source endpoint. Each READY allows the source to transmit one HIPPI burst (as explained below). All HIPPI source endpoints are required to be capable of enqueuing a minimum of 63 READYs. There is no minimum requirement for a destination's ability to generate READYs.[3] By sending ahead and enqueuing READYs, the two endpoints can optimize the throughput on their connection.

The source delineates its packets with the PACKET signal: at the beginning it asserts the PACKET signal, and at the end it deasserts the signal. A HIPPI packet consists of one or more bursts, as illustrated in Figure 1-4. Each burst contains 256 words, except in the case where the burst is short (as described below). The size of each word depends on the source's data bus (32 or 64 bits, as indicated by a bit in the I-field).[4] At the end of each burst, the source generates a checksum (LLRC) so that the destination can detect any errors in the received data; in addition, each word has four bits of parity for error checking.

Figure 1-4. HIPPI Packets and Bursts


The HIPPI protocol requires very small waiting periods between packets and between bursts. These required periods are counted in nanoseconds and are imperceptible to the user; however, in normal operation there may be noticeable pauses between bursts (for example, when the source is waiting to receive a READY).

As long as the source has READYs, it can transmit data as fast as it is capable of transmitting (but no faster than the protocol allows: 25 million words per second). When the sender has sent all the data for one packet, it indicates the end of the packet, using the PACKET signal. Indicating the end of the packet is necessary because HIPPI allows packet size to be undefined (indeterminate) at the start of the packet. A sender could essentially send an infinite–sized packet by keeping the PACKET signal asserted at all times.

A packet's first burst often contains some kind of header (for example, a HIPPI-FP header as described in “The FP Header”). The first burst can contain header only, or header and user data. In other words, the first words of user data can be in the first burst or the second. If the source program is generating HIPPI-FP packets, it can indicate the location of the packet's first word of user data by setting the B bit in the HIPPI-FP header.

Either the first or the last burst of a packet (but not both) can be less than 256 words. This burst is referred to as a short burst. Usually, the last burst is the short one. When the first burst is the short one, it contains only the header and, optionally, control information. The first word of the packet's user data is, in this case, located in the second burst, and the final burst may be padded to meet the 256-word length requirement. When the last burst is the short one, the packet's final burst never needs to be padded and the first word of user data may be included in the first burst.

Once the end of the packet has been indicated, the source has the option of keeping the connection open to transmit additional packets or of closing the connection.

Routing

The I-field contains HIPPI routing information in its 24-bit Routing Control field. This information is interpreted only by intermediate systems (switches); the Routing Control information does not need to be interpreted when the connection is directly between two endpoints.

The addresses in a Routing Control field can be in “logical addressing” or “source addressing” format. The format is indicated by the Path Selection bits of the I-field. The two formats cannot be used simultaneously in one I-field; however, both formats can be used simultaneously in one HIPPI network.


Note: The word source in “source addressing format” does not mean that the address is the source's address; it refers to the fact that the address, supplied by the source endpoint, defines the complete path (route).


Logical Addressing

With logical addressing, the Routing Control field contains two 12-bit addresses: a destination (receiver's) address and the source (sender's) address, as illustrated in Figure 1-5. The order in which the addresses are placed within the field is defined by the I-field's Direction bit, as illustrated in Figure 1-5.

Figure 1-5. Routing Control Field With Logical Addressing


When a HIPPI network uses logical addressing, each HIPPI network interface (endpoint) within the network is assigned an address that is unique within that network. One address can be used for both the source and destination channels of a network interface, if desired. Assignment of these addresses is a local matter; the addresses do not need to be unique outside the particular HIPPI network. Logical addresses have the formats described in Table 1-1. Some of the addresses are reserved for special purposes. 

Table 1-1. Logical Addressing Formats

Logical Address
(binary)

Number of Addresses

Usage

xxxx x0xx xxxx

4032

Endpoint addresses

1111 110x xxxx

32

Local assignment to network services

1111 111x xxxx

32

Reserved for global assignment

1111 1111 1111

1

Address is unknown

Each switch maintains a “map” of its network and uses a routing table to select the path along which to open a connection for each request. For example, Figure 1-6 illustrates a scenario where two paths are available between endpoints A and B. When endpoint A requests a connection to endpoint B, switch 1 can select either of these paths.

Figure 1-6. Routing With Logical Addressing


The 12 bits make it possible to create 4096 unique addresses. The HIPPI-SC standard reserves 64 of these addresses, leaving 4032 addresses available for local assignment to HIPPI end points. 4032 is the maximum number of destination endpoints that can exist on one HIPPI network using logical addressing.

Source Addressing

The addresses used for source addressing are of variable lengths, from 1 to 24 bits. When the Path Selection bits in the I-field indicate that source addressing is being used, the Routing Control field contains a list of port identifiers, as illustrated in Figure 1-7. The I-field's Direction bit determines the order in which the port identifiers are placed within the field and the alignment of (placement for) the addresses, as illustrated in Figure 1-7.

Figure 1-7. Routing Control Field (As Created by Sender) With Source Addressing


Each port identifier uniquely identifies one port within a switch. A port is a pair of physical links: both a source and a destination. For example, a 4x4 switch has 8 physical links to 4 systems, and for this it uses 4 port identifiers, as illustrated in Figure 1-8. Port identifiers are unique among all the ports on the same switch, but not among all the ports within the network. For example, a network with 5 switches might easily have 5 port identifiers of “1.” Figure 1-8 is an example of the port identifiers used in a network with 2 switches.

Figure 1-8. Switches and Port Identifiers


A Routing Control field in source address format is interpreted as a series of “stepping stones” leading to the destination in the following manner:

  1. The first switch (the one attached to the source endpoint) reads the first port identifier, opens a connection at that outgoing port, and sends the I-field (that is, the connection request).

  2. If the system at the end of that physical link is another switch, it reads the second port identifier, opens a connection at that outgoing port, and sends the I-field.

  3. And so on, until the receiving system is the destination endpoint.

When the port identifiers are followed sequentially, they create the path between the two endpoints. Each path (address) consists of a list of all the outgoing ports through which the connection request must pass in order to reach the destination. For example, in the simplest configuration, where one switch exists between two network interfaces, the address consists of one port identifier: the one to which the receiving interface is connected, as illustrated by Example 1 in Figure 1-9. When two switches exist between the interfaces, the address consists of two port identifiers, as illustrated by Example 2 of Figure 1-9.

Figure 1-9. Port Identifiers for Source Addressing


The Direction bit in the I-field defines whether each port identifier should be read from the most significant or least significant end of the Routing Control field. For example, Figure 1-10 illustrates two addresses that endpoint A might use to open a connection with B.

Figure 1-10. Routing With Source Addressing


Each HIPPI host within the network maintains a table of paths (addresses in source addressing format) for reaching each of the other endpoints. With each of its connection requests, a source attaches one of these paths, thus indicating how to reach the destination. The path is completely defined by the sending endpoint.

Unlike logical addresses (which are not altered enroute to the destination), addresses in source addressing format are changed by each switch that handles the I-field. The source program creates a list of outgoing port numbers that define a path from the sender to the receiver. By the time the packet arrives at its destination, the address has been altered so that it defines the return path (that is, the path from the receiver back to the sender). This change is brought about by each switch removing the outgoing port identifier that it reads, shifting the remaining bits into alignment, and adding an incoming port identifier (that is, the port through which the I-field just arrived), as illustrated in Figure 1-11.

Figure 1-11. How Switches Alter Source Addresses


A destination program can copy a received Routing Control field into its own I-field and simply change the setting of the Direction bit to open a return connection, thus bypassing the table lookup procedure. Normally, the source that first creates the Routing Control field sets the D bit to zero and places the address bits in the least significant positions of the Routing Control field. The receiver changes the setting for the D bit and uses the received Routing Control field exactly as it is received. In this manner, the port identifier labeled Last Incoming Port # in Figure 1-11 becomes the First Outgoing Port # for the return connection.

Port identifiers can be one to six bits in length. The number of bits varies from switch to switch. The size of the port identifier is the number of bits needed to uniquely identify all the possible ports on a switch. For example, a 4x4 switch has four ports and requires at least two–bit port identifiers (binary port identifiers 00, 01, 10, and 11). If a switch is capable of being enlarged, it may use large–sized port identifiers (for example, five or six bits) to avoid a reconfiguration of all the network's routing tables when the switch is upgraded.

The I-field's 24-bit Routing Control field limits the number of port identifiers that can be contained in an address, as summarized in Table 1-2.  

Table 1-2. Maximum Number of Port Identifiers in Routing Control Field

Number of Bits
Used in Port Identifier

Maximum Number of Port Identifiers Possible in
Routing Control Field

1

24

2

12

3

8

4

6

5

4

6

4


The Protocol

This section describes the format for the HIPPI I-field and FP header.

The I-field

The I-field is defined by the HIPPI-SC standard. The format for the 32-bit HIPPI I-field (also called CCI) is shown in Figure 1-12, and its fields are explained in Table 1-3.

Figure 1-12. I-field Format


Table 1-3. Fields of the HIPPI I-field

Field

Bits

Description

L

31

Local or Standard Format:

0=Bits 30:0 of I-field conform to the usage described in this table.

1=Bits 30:0 are implemented in conformance to a private (locally-defined) protocol.

VU

30:29

Vendor Unique Bits:

Vendors of end-system HIPPI equipment may use these bits for any purpose. Switches do not alter or interpret these bits.

W

28

Width:

0=The data bus of the transmitting (source) HIPPI is 32 bits wide for 800 megabits/second transmission.

1=Source's data bus is 64 bits wide for 1600 megabits/second transmission.

D

27

Direction:

0=Least significant bits of Routing Control field contain the destination address for the current switch to use.

1=Most significant bits of Routing Control field contain the destination address for the current switch to use.

PS

26:25

Path Selection:

00=Source routing.

01=Logical routing. Switch must select first route from a list of routes.

10=Reserved.

11=Logical routing. Switch selects any (or best) route from its list.

C

24

Camp-on:

0=Switch rejects connection request immediately if port to destination is busy.

1=Switch holds connection request if port to destination is busy and establishes connection when the port becomes free or when source aborts the request.

Routing
Control

23:0

Address:

This 24-bit field contains addressing/routing information. The contents are in source routing or logical routing format, as indicated by the PS field.

For source routing, the field contains a list of switch port identifiers that, when followed, lead to the destination.

For logical addressing, the field contains two 12-bit addresses (receiver's and sender's) that are used by the intermediate switches to select a route from a table.


The FP Header

The FP header is defined by the HIPPI-FP standard. When a HIPPI endpoint is HIPPI-FP conformant, all the packets it transmits and/or receives (without error) are HIPPI-FP packets. The first burst of each of its transmitted packets contains an FP header, and it looks for an FP header in the first burst of each received packet. A HIPPI-FP packet consists of three segments, listed below and illustrated in Figure 1-13. Each segment is eight-byte aligned (that is, contains an integral number of 64-bit words).

  • Framing Protocol header:
    This area contains the 64-bit HIPPI-FP header, described in more detail in Table 1-4.

  • D1_Area:
    This optional area, if present, must be completely contained in the first burst. It may contain control information (the D1 data set), it may be defined for padding purposes only, or it may serve both of these functions. The D1 area can be 0 to 255 words in size; however, regardless of the word size used by a HIPPI implementation, the D1 area must contain an integral number of 64-bit words. So, for a 64-bit implementation, the D1 area can have up to 255 words. For a 32-bit implementation, the D1 area can have up to 254 words.

    The D1 data set (located within the D1 area) is optional. The maximum size of any D1 data set is 1016 bytes (that is, 254 32-bit words), thus allowing the FP header (8 bytes) and D1 data to fit in the first burst of any HIPPI implementation (for example, a burst made of 32-bit words). The content and format of the D1 data is locally defined and the data must be self-defining. For example, each upper layer application (with its own ULP-id) could use a different format and length for its D1 data.

  • D2_Area:
    The optional D2 area contains the user/application data. This area can be 0 to 4-gigabytes minus 1-byte in size, or it can be defined as indeterminate. The size of this area must be an integral number of 64-bit words. The area may contain padding (an offset and possibly filler).

    Figure 1-13. HIPPI-FP Packet Format


The 64-bit FP header describes the HIPPI packet using six fields, illustrated in Figure 1-14 and described in Table 1-4.

Figure 1-14. FP Header Format


Table 1-4. Fields of the HIPPI-FP Header

Field

Bits

Description

ULP-id

63:56

The 8-bit upper layer protocol identification field identifies a system's upper layer protocols. A transmitting application uses this number to specify the upper layer protocol of the intended recipient of the packet. A receiving HIPPI subsystem can use this number to demultiplex incoming packets among a number of upper layer protocols (or applications) and to determine whether an intended recipient is known or not.

P bit

55

The 1-bit present bit indicates whether or not the packet contains D1 data. Note that the D1_Area may be present (defined by the D1_Area Size field), but empty (the P bit is set to 0). 0=no D1 data for this packet; 1=D1 data exists for this packet.

B bit

54

The 1-bit burst boundary bit indicates which burst contains the first byte of D2 data. D2 data can be included in the first burst or it can start with the first word of the second burst. 0=there is D2 data in the packet's first burst; 1=D2 data starts on first byte of second burst.

D1 Area Size

42:35

The 8-bit D1 area size field indicates the number of 64-bit words in the D1_Area of this packet. The area does not necessarily contain valid data; that is, the area may be defined for padding purposes only. Note that the size is always stated in 64-bit words, regardless of the implementation's word size.

D2 Offset

34:32

The 3-bit D2 offset field indicates the number of bytes between the last byte of D1 data and the first byte of D2 data. This field is used only when D2 data is present in the first burst.

D2 Data Size

31:0

The 32-bit D2 data size field indicates the number of bytes of D2 data included in this packet. Bytes of offset or fill are not included in this count.

Figure 1-15 illustrates a HIPPI-FP packet. The first burst of this packet (words 0 to 255) contains only the FP header and the D1 data. The D1 area in this packet is completely full of D1 data. Notice that the P bit in the FP header is set to 1 to indicate the presence of valid D1 data. The D2 data starts on word 256, the first byte of the second burst, as indicated by the B bit setting.

Figure 1-15. Sample HIPPI-FP Packet


Figure 1-16 illustrates some of the HIPPI-FP packets that are commonly created by applications. Notice how example 2 uses an empty D1_Area to pad out the first burst, thus locating the data in the second burst. Examples 2, 3, and 4 all illustrate this technique. Also notice, examples 1 and 5, how user (D2) data can be placed in the first burst. with or without D1 data.

Figure 1-16. Some Common HIPPI-FP Packets


The Signals

The signals at the HIPPI-PH layer of each physical link are used for controlling the connection, packet boundaries, and data flow. These signals are described in Table 1-5 and illustrated in Figure 1-17. Within the figure, the signals are numbered to represent the order in which they are asserted when power is first applied at the two endpoints. The two INTERCONNECT signals are not dependent on any other signal; they are asserted as the HIPPI hardware receives power and becomes active. Each of the other signals is asserted only when all the signals before it have been observed.

Figure 1-17. HIPPI Signals Used on Each Point-to-Point Connection


Table 1-5. HIPPI Signals

SIGNAL

DESCRIPTION

Generated by the source on this physical link

 

Source-to-Destination

INTERCONNECT

When asserted, indicates source is attached and ready for action. This signal is sometimes referred to as SDIC.

REQUEST

When asserted, indicates source is requesting a connection to be opened. This signal is accompanied by an I-field.

When deasserted, indicates the source is closing the connection (if one is open) or aborting the connection request (if no connection is currently open).

PACKET

When asserted, indicates a packet is in progress.

When deasserted, indicates the end of a packet.

This signal does not indicate that any data is being sent; it only delineates the boundaries of a packet.

BURST

When asserted, indicates data is being sent. This signal is accompanied by one burst of data.

When deasserted, no data is being sent.

Generated by the destination on this physical link

 

Destination-to-Source

INTERCONNECT

When asserted, indicates destination is attached and ready for action. This signal is sometimes referred to as DSIC.

CONNECT

When asserted, indicates destination is accepting the connection (opening a connection in response to a REQUEST signal).

When deasserted, indicates destination is closing the connection (if one is open) and is now available for a new connection.

READY

When pulsed, indicates destination can accept (has buffer space available) one burst of data. Destination can send any number of these to source; however, the source is required by the HIPPI-PH standard to queue only 63.


HIPPI Configurations

This section describes some of the common configurations of HIPPI equipment. Because HIPPI is a simplex point-to-point protocol (or in the case of HIPPI–Serial, dual–simplex), only one source can transmit user (upper–layer) data onto the transport medium (cable) between the two endpoints. Two physical links are required for bidirectional communication. This aspect of HIPPI makes it quite different from protocols such as Ethernet, FDDI, or 802.5 Token Ring.

Basic HIPPI Configurations

A basic HIPPI configuration consists of two endpoints, one sending and the other receiving user data, as shown in Figure 1-18.

Figure 1-18. Basic HIPPI Configuration


To exchange data in both directions, two physical links and two connections are required between the two endpoints.[5] Each endpoint's source channel must open a connection with the destination of the other endpoint. Some HIPPI products (for example, all Silicon Graphics' HIPPI products) allow each simplex connection to connect to a different endpoint, as illustrated by the two examples on the right in Figure 1-19.

Figure 1-19. Three Variations of the Basic Configuration


The copper-based IRIS HIPPI mezzanine board for the CHALLENGE and Onyx platforms has two ports: one provides the source physical link and the other provides the destination physical link. The fiber-optics based IRIS HIPPI–Serial XIO™ board for the Origin series and Onyx2 platforms has one port that supports the two simultaneous connections in opposite directions. The IRIS HIPPI software treats each link as a separate entity, so that each IRIS HIPPI board can support two autonomous, simultaneous connections (one sending and one receiving). The two connections can be to two different endpoints (as shown by the two examples on the right in Figure 1-19) or to the same endpoint (as illustrated by the example on the left in Figure 1-19). IP communication over HIPPI requires the latter configuration.

HIPPI Network Configurations

One or more HIPPI switches may be placed along the endpoint-to-endpoint link, making it possible to configure a number of endpoints into a HIPPI network (or fabric). Configuring the endpoints in this way does not alter the fact that each communication is a point-to-point connection. The switches are cross switches that may include demultiplexing functionality; but they are not “routers.”

When a switch is included in a HIPPI configuration, each endpoint has a number of hosts with which it can communicate (one at a time). Figure 1-20 illustrates a HIPPI network with one switch. The switch in this illustration is a 4 x 4, meaning that the switch can have four systems (8 HIPPI links) attached to it. The switch supports four simultaneous connections. For example, in Figure 1-20, any one of the following connection scenarios could be occurring at any single point in time:

  • A and D could be exchanging TCP/IP traffic. There would be two connections open between them. C and D could be doing the same. This scenario opens all four possible connections.

  • A could be transmitting to B, while B transmitted to C, C to D, and D to A. This scenario also opens four connections.

  • A and C could be exchanging bidirectional traffic. D could be transmitting to B. Only three connections are open in this scenario.

    Figure 1-20. HIPPI Network Configuration With One Switch


Figure 1-21 illustrates a network with multiple switches, and Figure 1-22 illustrates a complex HIPPI network including a long-distance fiber optic link and multiple ports between switches to improve connection setup time by reducing the probability of encountering a busy link.

Figure 1-21. HIPPI Network Configurations With Multiple Switches


Figure 1-22. Complex HIPPI Network Configuration


The maximum number of switches and endpoints within a network is limited by three factors:

  • When logical routing is used, the 12-bit HIPPI address (half of the Routing Control field) limits the number of unique endpoint addresses to 4096. It is possible for a site to implement this number of networked HIPPI endpoints; however, to be compliant with the HIPPI-SC standard, 64 reserved addresses should not be assigned to local endpoints (that is, hosts). This limits the number of endpoints to 4032 per network. There is no limit to the number of switches when logical routing is used.

  • When source routing is used, the I-field's 24-bit Routing Control field limits the number of port identifiers that can be included in the list. The exact number depends on the sizes of the port identifiers used by the switches along the specific endpoint-to-endpoint path, as explained in “Source Addressing.” Each port identifier in the Routing Control field represents one switch along the path. Table 1-6 summarizes the maximum number of switches along any point-to-point path within a HIPPI network, assuming that all switches along that path use port identifiers of the same size. (This assumption does not reflect actual site configuration practices, but is useful here for illustration of a point.) When source routing is used, the number of switches and endpoints that are possible is not limited; however, the number of switches between any two endpoints within the network is limited. This limit affects the configuration of the network.

  • If a network is built according to the guidelines in Appendix B of RFC 1374, the recommended maximum number of hops (switches) between any two endpoints is three. This limit has major implications for the structure and size of a HIPPI network. The structure is limited to a single hub switch with satellite switches attached to the hub's ports, but no switches attached to any satellite ports. Figure 1-21 shows an example of an RFC 1374–compliant network. Table 1-7 summarizes the maximum number of switches and endpoints possible for a network in which switches of only one size are used throughout the network.

Table 1-6. Maximum Number of Switches Along Any Single Point-to-PointPath When Using Source Addressing

Number of Bits Used for All Port IDs in Routing Control Field

Max. Number of Switches Along Any Single Point-to-Point Path

1

24

2

12

3

8

4

6

5

4

6

4


Table 1-7. Maximum Number of Switches and Endpoints on a Network Built in Accordance With RFC 1374, Appendix B Guidelines

Size of All Switches Within Network

 

 

 

 

4 x 4

8x 8

16 x 16

32 x 32

64 x 64

5 switches /
12 endpoints

9 switches /
56 endpoints

17 switches /
240 endpoints

33 switches /
992 endpoints

65 switches /
4032 endpoints


The HIPPI Standards and Documentation

The documents listed below provide the official definitions of what HIPPI is and how it works.

  • ANSI HIPPI-PH
    The HIPPI Mechanical, Electrical, and Signalling document defines the standard for a copper physical layer: electrical and mechanical aspects of copper HIPPI cables, connectors, transmitters, and receivers. This document defines the HIPPI signals (like SDIC, DSIC, REQUEST, CONNECT, READY, PACKET, and BURST) that are used in all HIPPI implementations, including HIPPI-Serial.

  • ANSI HIPPI–Serial
    The High–Performance Parallel Interface–Serial Specification: X3.300-1997 document defines the ANSI standard for a fiber-optic physical layer: electrical and mechanical aspects of fiber–optic cables, connectors, transmitters, and receivers. The method for communicating (encoding) the HIPPI–PH signals in serial is also defined.

  • ANSI HIPPI-SC
    The HIPPI Physical Switch Control document defines the standard for switch behavior, routing methods, and connection management. The HIPPI I-field is defined by this standard.

  • ANSI HIPPI-FP
    The HIPPI Framing Protocol document defines the standard for data framing issues: how a packet is formed, how its data contents are described and interpreted. The HIPPI-FP packet (FP header, D1_Data, and D2_Data) is defined by this standard.

  • ANSI HIPPI-LE
    The HIPPI Encapsulation of ISO 8802-2 (IEEE 802.2) Logical Link Control Protocol Data Units (HIPPI-LE) standard defines the method for encapsulating (and thus interoperating with) 802.2 compliant data link layers such as FDDI, 802.5 Token Ring, and CSMA/CD (Ethernet).

  • ANSI HIPPI-IPI-3 for Disk
    The HIPPI Intelligent Peripheral Interface—Device Generic Command Set for Magnetic and Optical Disk Drives standard defines an upper–layer protocol for interfacing disks to the HIPPI subsystem.

  • ANSI HIPPI-IPI-3 for Tape
    The HIPPI Intelligent Peripheral Interface—Device Generic Command Set for Magnetic Tape Drives standard defines an upper–layer protocol for interfacing tapes to the HIPPI subsystem.

  • RFC 1374
    IP and ARP on HIPPI, by J. Renwick and A. Nicholson (October 1992) defines the protocol for using the IP suite of network and transport layer protocols over HIPPI.

The ANSI documentation for HIPPI standards is maintained by the American National Standard of Accredited Standards Committee (ANSI X3T9.3). Information about obtaining copies of the ANSI standards listed above is provided at the HIPPI Network Forum's webpage (URL): http://www.esscom.com/hnf//html/hippispecs.html. Or, call ANSI at telephone 212-642-4900 in New York, USA.

Implementation Details for IRIS HIPPI

This section describes some of the details of the Silicon Graphics implementation of the HIPPI protocol.

How HIPPI Boards Are Assigned to Interfaces

This section describes the manner in which IRIX assigns an IP network interface (for example, hip0 and IP address 223.9.1.2) to a particular HIPPI port.

On CHALLENGE and Onyx Platforms

On a CHALLENGE or Onyx system, with each restart (for example, after a reboot, shutdown, halt, or init command, or a power off), the startup routine probes for hardware installed in the mezzanine I/O adapter slots, and makes a list (inventory) of all the boards located. The slots are probed in the following order:

  • main IO4 board: I/O adapter slot 5, then 6

  • second IO4 board (if present): I/O adapter slot 2 (only when the FMezz board is long), slot 5, slot 3 (only when the FMezz board is long), slot 6

  • third IO4 board (if present): I/O adapter slot 2 (only when the FMezz board is long), slot 5, slot 3 (only when the FMezz board is long), slot 6

  • fourth IO4 board (if present): I/O adapter slot 2 (only when the FMezz board is long), slot 5, slot 3 (only when the FMezz board is long), slot 6

The list and order of IRIS HIPPI boards that were located by this process can be displayed with the /sbin/hinv command, as shown below. The text hippi# indicates the order: hippi0 is the first board located and hippi1 is the second. In this example, the startup routine located two IRIS HIPPI boards attached to FMezz boards on two different IO4 boards.

% /sbin/hinv -d hippi
...
HIPPI adapter: hippi0, slot 5 adap 6, firmware version ####
HIPPI adapter: hippi1, slot 3 adap 5, firmware version ####

As the startup process begins to initialize HIPPI network interfaces, it does the following:

  • If the IRIS HIPPI driver is configured to support the IP protocol stack, the driver creates a network interface for each HIPPI board in the inventory. The first HIPPI interface (always named hip0) is associated with the first board listed in the inventory; the second interface (hip1) is associated with the second board on the list; and so on, until there are no more boards. For example, using the configuration shown in the hinv example above, interface hip0 is assigned to board hippi0 and hip1 is assigned to board hippi1.

  • The ifconfig command (which is invoked automatically during startup) searches the netif.options file for IP over HIPPI interface names (for example, hip0, hip1, hip2) and configures and enables each interface that exists (that is, each interface that was created by the driver).


Note: If an installed board is not located due to a loose connection or malfunction, or if hardware is installed or removed, the assignment of HIPPI network interfaces to boards may change. For example, hip0 (from the example above) could be assigned, at a later reboot of the machine, to the hippi1 board instead of hippi0, if hippi0 were not found.


On Origin and Onyx2 Platforms

On an Origin2000, OriginServer, or Onyx2 system, with each restart (for example, after a reboot, shutdown, halt, init command, or a power off), the startup routine probes for hardware on all the systems connected into the CrayLink™ interconnection fabric. All the slots and links in all the modules within the fabric are probed. The routine then creates a hierarchical file system, called the hardware graph, that lists all the hardware that is located. The top of the hardware graph is visible at /hw. (For complete details, see the reference page for hwgraph.) After the hardware graph is completed, the ioconfig program assigns a unit number to each located device that needs one. Other programs (for example, hinv and the device's driver) read the assigned number from ioconfig and use it. On an initial startup, ioconfig assigns numbers sequentially; for example, if three IRIS HIPPI boards are found, they are numbered unit0, unit1, and unit2. On subsequent startups, ioconfig distinguishes between hardware that it has seen before and new items. To previously seen items, it assigns the same unit number that was assigned on the initial startup. To new hardware, it assigns new sequential numbers. It never reassigns a number, even if the device that had the number is removed and leaves a gap in the numbering.


Note: New items are differentiated from previously-seen items based solely on the hardware graph listing (that is, the path within /hw). The database of previously-seen devices is kept in the file /etc/ioconfig.conf. For example, a new replacement board that is installed into the location of an old board will be assigned the old board's number, while a board that is moved from one location to another will be assigned a new number. For more information about the hardware graph and ioconfig, see the reference (man) pages for hwgraph and ioconfig.

The IRIS HIPPI boards that are located can be displayed with the /sbin/hinv or find commands, as shown below. In these examples, the startup routine located two IRIS HIPPI boards on two different modules (that is, inside two different chassis).

% find /hw/module -name hippi
/hw/module/1/slot/io3/hippi_s/pci/0/hippi
/hw/module/2/slot/io12/hippi_s/pci/0/hippi

% /sbin/hinv -d hippi
HIPPI-Serial adapter: unit 0, in module 1 I/O slot 3
HIPPI-Serial adapter: unit 1, in module 2 I/O slot 12

As the startup process continues, it calls the network hardware drivers so that they can create their network and programmatic interfaces. For HIPPI, this step works in the following manner:

  • For each IRIS HIPPI board, the installation script creates a symbolic link in /dev that points to the board's entry in the hardware graph. Subsequently, the driver creates short (/hw/hippi/#) and long (/hw/module/#/slot/.../hippi.) entries in the hardware graph. The /dev/hippi# links are for use by the IRIS HIPPI application programming interface (API).

  • If the driver is configured to support the IP protocol stack, the driver creates an IP network interface for each board. The network interface number always matches the board's assigned unit number. For example, if the only IRIS HIPPI board found during startup is known by ioconfig as unit2, then the driver creates only network interface hip2.

  • The ifconfig command (which is invoked automatically during startup) searches the netif.options file for IP–over–HIPPI interface names (for example, hip0, hip1, hip2) and configures and enables each one that has been created.


Note: The assignment of network interfaces to boards does not change across restarts. If a board that was once known to the system is not located during a reboot, the network interface that matches that board is not created. For example, if IRIS HIPPI unit 1 is not found, the hip1 network interface is not created; the unit2 and hip2 pairing proceeds as normal.


Site Cabling

IRIS HIPPI for Challenge and Onyx platforms is copper based (HIPPI-PH). Each I/O panel plate has two 100–pin connectors. For each IRIS HIPPI board, the site must provide two cables of up to 25 meters each.

IRIS HIPPI for OriginServer and Onyx2 platforms is fiber based (HIPPI-Serial). Each I/O panel plate has one dual-SC receptacle. For each IRIS HIPPI board, the site must provide one fiber-optic cable of either 50–micron core (2 to 500 meters in length) or 62.5–micron core (2 to 200 meters). The IRIS HIPPI-Serial board uses short wavelength (770-860 nanometer) optics. The cable should be terminated with a dual-SC connector; however, 2 simplex SC connectors will also work. (Figure 1-23 illustrates the data direction used in the board's dual–SC receptacle.) Each cable must conform with the HIPPI-Serial physical layer specification and guidelines, as described in the ANSI High–Performance Parallel Interface–Serial Specification, version 2.6, especially annex B and the section entitled “Serial Optical Interface.” 62.5-micron core cables in various lengths (including a loopback cable for testing) can be purchased from Silicon Graphics, as summarized in Table 1-8. In addition, the site cabling throughout the HIPPI switch fabric must conform to this specification. Table 1-9 summarizes some of this specification's information.

Figure 1-23. Data Direction Used in IRIS HIPPI-Serial Dual–SC Receptacle


Table 1-8. Fiber Optic Cables Available from Silicon Graphics

Description

Part Number

3-Meter Fiber Optic Cable Assembly

018-0656-001, X-F-OPT-3M

10-Meter Fiber Optic Cable Assembly

018-0656-101, X-F-OPT-10M

25-Meter Fiber Optic Cable Assembly

018-0656-201, X-F-OPT-25M

100-Meter Fiber Optic Cable Assembly

018-0656-301, X-F-OPT-100M

Fiber Optic Loopback Test Cable Assembly

018-0664-001, X-F-OPT-LOOP


Table 1-9. Specifications and Limits for HIPPI-Serial Optics

Item

Value for 50 micron cable

Value for 62.5 micron cable

Cable

 

 

 minimum bandwidth at 780 nm

500 MHzkm

160 MHzkm

 maximum loss

4 dB/km

4 dB/km

Connectors

 

 

 maximum number between 2 nodes

8

8

 mean loss per unit

≤0.11 dB

≤0.11 dB

 standard deviation loss per unit

0.15 dB

0.15 dB

 minimum optical return loss

20 dB

20 dB

Splices

 

 

 maximum number between 2 nodes

6

6

 mean loss per unit

≤0.08 dB

≤0.08 dB

 standard deviation loss per unit

0.05 dB

0.05 dB


Application Programming Interface

The IRIS HIPPI driver provides access and control of the HIPPI subsystem to upper-layer applications. The upper layer-applications that are shipped with the IRIS HIPPI product are the IRIS HIPPI-LE module serving the IP network stack, and the IRIS HIPPI utilities. Upper-layer programs can also be developed by customers, using the IRIS HIPPI application programming interface.

Customer-developed applications can define their own upper-layer protocol (ULP) and can program the IRIS HIPPI driver to implement HIPPI-FP, or they can bypass the HIPPI-FP layer and access the HIPPI protocol directly at the HIPPI-PH layer. Refer to the IRIS HIPPI API Programmer's Guide for complete details.

The rest of this section describes the IRIS HIPPI-LE upper-layer protocol implementation.

Handling of HIPPI Protocol for HIPPI-LE

This section describes how the IRIS HIPPI driver and the HIPPI-LE module handle HIPPI I-fields, HIPPI-FP headers, and 802.2 encapsulation items. There are separate sections for transmission and reception.

On Transmission

The HIPPI-LE module obtains the I-field for each destination from a lookup table that is initialized at startup time from the /usr/etc/hippi.imap file. It obtains this I-field value before programming the IRIS HIPPI driver to make a connection request. The hippi.imap file maps IP addresses (or host names) to 32-bit values that are used as I-fields. The software uses each I-field value exactly as read from the file, as summarized in Table 1-10. It is the responsibility of the system administrator to ensure that the values in the lookup table are appropriate for the site's configuration.

Table 1-10. I-field Recommended for Use With IRIS HIPPI-LE



Field

Value Recommended
for HIPPI-LE



Comments

L

0

0=HIPPI-SC compliant; 1=local format for I-field.

A site may use any value.

VU

0

A site may use any value.

W

0

0=32 data bus.
No other setting is supported by the IRIS HIPPI board.

D

0

0=Least significant bits contain address for next hop; 1=address is placed in most significant bits.
A site may use any setting.

PS

any setting

A site may use any of the addressing formats.

C

1

1= camp-on; 0=do not camp-on.
This setting makes the HIPPI network more efficient; however, a site may use any setting.

Routing
Control

any setting

A site may use any settings.

Once the connection has been opened, the HIPPI-LE module creates a HIPPI packet in the format illustrated in Figure 1-24. This packet conforms with the HIPPI-FP standard and the RFC 1374 guidelines.

Figure 1-24. HIPPI Packet Created by IRIS HIPPI-LE


The IRIS HIPPI-LE module creates the HIPPI-FP header with the values summarized in Table 1-11. 

Table 1-11. FP Header Created by IRIS HIPPI-LE ULP

Field

Value Used

Comments

ULP-id

4 = HIPPI-LE

As defined by HIPPI-FP.

P bit

1 = D1 area is included in this FP header

D1 area contains the HIPPI-LE header as defined by HIPPI-LE.

B bit

0 = D2 data is included in first burst

As specified by RFC 1374.

D1 Size

3 = three 64-bit words (that is, 24 bytes)

As defined by HIPPI-LE.

D2 Offset

0

 

D2 Size

Up to 64 kilobytes.

Maximum IP packet as defined by the Internet Protocol.

The IRIS HIPPI-LE module creates the HIPPI-LE header with the values summarized in Table 1-12. The HIPPI-LE header becomes the D1 data set for the HIPPI packet. 

Table 1-12. D1 Data (HIPPI-LE Header) Created by IRIS HIPPI-LE

Field

Size

Value Used

Comments

FC

3 bits

0

As defined by HIPPI-LE and restated in RFC 1374.

Double Wide

1 bit

0 = 32 bit data bus

 

Message Type

4 bits

0 = data

 

Destination Switch Address

24 bits

0

 

Destination Address Type

4 bits

0

 

Source Address Type

4 bits

0

 

Source Switch Address

24 bits

0

 

Reserved

16 bits

0

 

Destination IEEE Address

48 bits

0

 

LE Locally Administered

16 bits

0

 

The IRIS HIPPI-LE module creates the 802.2 headers (LLC and SNAP) with the values summarized in Table 1-13. This information occupies the initial bytes of the D2 data within the HIPPI packet. 

Table 1-13. IEEE 802.2 Header (First Bytes of D2) Created by IRIS HIPPI-LE

Field

Size
(bits)

Value Used

Comments

SSAP

8

170 decimal

As defined by IEEE 802.2 standard for Logical Link Control and restated in RFC 1374.

DSAP

8

170 decimal

Same as above.

CTL

8

3

Same as above.

Organization Code

24

0

Same as above.

EtherType

16

2048 decimal

Same as above.


On Reception

The IRIS HIPPI driver accepts all connection requests, and accepts all packets containing FP headers with known ULP-ids, thus supporting customer-developed, upper-layer applications. The I-field for the connection request is not interpreted. This process is summarized in Table 1-14. 

Table 1-14. I-field Accepted by IRIS HIPPI Driver for HIPPI-LE ULP

Field

Recommended Values

Comments

L

0 = HIPPI-SC compliant

Content is ignored.

VU

any value

Content is ignored.

W

0 = 32 data bus

Content is ignored.

D

any value

Content is ignored.

PS

any value

Content is ignored.

C

any value

Content is ignored.

Routing
Control

any value

Content is ignored.

Once a connection has been opened, the IRIS HIPPI driver places each incoming HIPPI packet on the input queue for the ULP-id indicated in the FP header. Incoming HIPPI packets must have the format illustrated in Figure 1-25. If the ULP-id is not known to the driver, the packet is dropped (that is, accepted then discarded). Packets with a ULP-id of 4 are enqueued for the HIPPI-LE module.

Figure 1-25. HIPPI Packets that IRIS HIPPI Driver Passes to HIPPI-LE


The HIPPI driver interprets only the FP header. All further processing of the HIPPI packet (including the various protocol headers) is done by the reader of the input queue (for example, HIPPI-LE).

On reception, the IRIS HIPPI driver handles HIPPI-FP headers as summarized in Table 1-15. 

Table 1-15. FP Header Accepted by IRIS Driver for HIPPI-LE ULP

Field

Values Accepted Without Generating an Error

Comments

ULP-id

4 = HIPPI-LE

As defined by HIPPI-FP. For other ULP-ids, see the IRIS HIPPI API Programmer's Guide for details.

P bit

1

If set to 1, the driver interprets the D1 area as a HIPPI-LE header.
If set to 0, the packet is discarded.

B bit

any value

For applications using the HIPPI-FP access method, the IRIS HIPPI driver passes the D1 data to the input queue reader as a separate item from the D2 data.

D1 Size

3 = three 64-bit words /
   24 bytes

As defined by HIPPI-LE.
If the value is different, the packet is discarded.

D2 Offset

any value

 

D2 Size

up to 64 kilobytes

As defined by Internet Protocol.
If size is greater than 64 kBytes, the packet is discarded.

The IRIS HIPPI-LE upper layer program handles received D1 data (the HIPPI-LE header) as summarized in Table 1-16. 

Table 1-16. D1 Data Accepted by IRIS HIPPI-LE ULP

Field

Values Accepted Without Generating an Error

Comments

FC

0

As defined by HIPPI-LE.
If the value is different, an error is generated.

DW

0 = 32 bit data bus

If set to 1, an error is generated.

MT

0 = data

If not 0 (data), an error is generated.

Dest_Sw_Addr

any value

 

Dest_Addr_Type

any value

 

Src_Addr_Type

any value

 

Src_Sw_Addr

any value

 

Dst_IEEE_Addr

any value

 

LE_Locally_Adm

any value

 

Src_IEEE_Addr

any value

 

The IRIS HIPPI-LE module also handles the 802.2 headers as summarized in Table 1-17. 

Table 1-17. IEEE 802.2 Headers Accepted by HIPPI-LE ULP

Field

Size
(bits)

HIPPI-LE Default

Comments

SSAP

8

170 decimal

As defined by the IEEE 802.2 standard and restated by RFC 1374. If the received value is different, an error is generated.

DSAP

8

170 decimal

Same as above.

CTL

8

3

Same as above.

Organization Code

24

0

Same as above.

EtherType

16

2048 decimal

Same as above.




[1] IRIS HIPPI supports only 800 megabits per second.

[2] IRIS HIPPI–Serial uses shortwave optics that can support cable lengths of up to 500 meters.

[3] The source channel on the IRIS HIPPI for Challenge/Onyx board can enqueue up to 65,535 READYs; the destination channel can generate up to 255 outstanding READYs. Each channel of the IRIS HIPPI–Serial XIO board can handle 128 READYs.

[4] The IRIS HIPPI board supports 32-bit words only.

[5] In HIPPI-Serial implementations, each fiber-optic cable attached to a dual-SC port contains 2 links.