The GSN products offered by SGI consist of multiple components that implement the following protocols:

The SGI GSN hardware and IRIX GSN software support the following network stacks (illustrated in Figure 1-1):

IRIX GSN complies with the following industry standards:

The following strings are used to identify the GSN product:

IRIX GSN 2.1 requires IRIX 6.5.9m for TCP/UDP and 6.5.9f for full TCP/UDP/STP support. Use the versions command to verify the version of IRIX that is currently running on the system. The version number (indicated by the -n option) must be equal to or greater than the version shown in the following example:

% versions -n eoe 
I eoe 1275719131  IRIX Execution Environment, 6.5.9m

The SGI GSN hardware requires the system's HUB ASICs to be version 5. Use this command to verify the version of the HUB on each Node board:

% hinv -v | grep HUB
HUB in Module #/Slot 1: Revision 5 Speed 97.50 Mhz (enabled)
HUB in Module #/Slot 2: Revision 5 Speed 97.50 Mhz (enabled)
HUB in Module #/Slot 3: Revision 5 Speed 97.50 Mhz (enabled)
HUB in Module #/Slot 4: Revision 5 Speed 97.50 Mhz (enabled)

The following terms have specific meanings when used within the context of GSN: 

The GSN protocol provides 6.4 gigabits of user data per second from source to destination (in each direction) over either copper-based or fiber–optics-based physical media.^[1] The protocol is point-to-point, full-duplex, and flow-controlled. It uses small fixed-size micropackets (illustrated in Figure 1-4 and Figure 1-6) and up to four interleaved logical datastreams (channels) per point–to–point connection.

Each physical link is composed of two simplex links that connect two HIPPI-6400 elements; data flows in only one direction on each simplex link. Both simplex links are required for a connection because control information for each datastream travels in the reverse direction (that is, along the other simplex link of the connection). This design provides a full-duplex connection between two endpoints.

The GSN data rate is stated as 6.4 gigabits of user data per second on each simplex link; however, each link physically carries a total of 8 gigabits (1 gigabyte) of data (user and control) every second. The following items describe the GSN bandwidth:

Each simplex link can carry up to four logical datastreams ( virtual channels). These virtual channels are allocated for control traffic, low latency traffic, and bulk traffic to avoid the latency/blocking issues that occur when only a single channel is attempting to handle both bulk and interactive traffic.

Each virtual channel is commonly implemented as a queue; micropackets are selected alternately from the active queues and placed onto the physical link in an interleaved fashion, as illustrated in Figure 1-2. Not all four channels need to be active on every connection. All the micropackets belonging to a single GSN Message always travel through the same channel, even when the message traverses switches along its way to the final destination. The restrictions for the data that can be carried on each channel are described in Table 1-3. 

Figure 1-2. GSN Micropackets from Virtual Channels Interleaved in Datastream

The micropacket is the basic protocol data unit for GSN. Each GSN micropacket is 32-bytes of data accompanied by 8 bytes (64 bits) of control information. The TYPE field within the control bits indicates the format and purpose of the micropacket's 32 bytes of data. The VC field determines which virtual channel carries the micropacket. Some of the control bits that accompany a 32-byte chunk of data refer to that chunk of data (for example, the VC and TYPE fields), and some bits refer to the datastream traveling in the opposite direction on the other physical link (for example, the credits in the CR field that allow the reader/receiver of the control bits to transmit more data for its own datastream). Figure 1-3 illustrates the control bits and Table 1-4 describes them. Table 1-5 summarizes the different TYPEs of GSN micropackets.

Figure 1-3. GSN Micropacket Control Bits

Most of the GSN micropacket TYPEs are related to control and management of the GSN link. Only three TYPEs of micropackets are passed to the upper layers: Admin, Header, and Data micropackets. The Admin micropacket (illustrated in Figure 1-4) is used by upper-layer GSN administrative programs to manage and configure a GSN fabric; hence, the Admin micropacket is defined by the Switch Control ANSI standard (HIPPI-6400-SC). The Header and Data micropackets are used to create GSN Messages (illustrated in Figure 1-6) that carry user-level data. 

One of the functions for the Admin micropacket is to allow each switch on a GSN fabric to discover the fabric's physical configuration and each endpoint to discover the universal LAN MAC address (ULA) that its switch has assigned to it. This functionality is not available on every GSN product; however, when it is implemented, this is how it works.

A GSN destination (receiving) endpoint controls the flow of micropackets by periodically releasing credits to the source.^[2] Each credit represents memory at the destination for one GSN micropacket. Each credit gives the source permission to send one additional micropacket on a specific channel. The destination gives credits to the source in the control bits (CR and VCR bits) that accompany the destination's own micropackets. Note that the credits travel in the opposite direction from the data, as illustrated in Figure 1-5, and can accompany micropackets traveling on any of the GSN virtual channels for the connection.

Figure 1-5. GSN Flow Control

The GSN Message is the basic data transfer unit between source and final destination endpoints. Each Message is composed of one initial Header micropacket followed by zero or more Data micropackets (illustrated in Figure 1-6). The micropackets of a Message are sequentially ordered and all travel over the same virtual channel using the same originating source (S_ULA value) and final destination (D_ULA value). The last micropacket in a Message has a bit set (the TAIL flag) to indicate that the Message is complete. Figure 1-6 illustrates a complete GSN Message.

Figure 1-6. GSN Message Composed of Header and Data Micropackets

When the GSN Header micropacket is carrying an IP datagram (EtherType=0x0800), the 8 bytes of payload in the Header micropacket are the first 8 bytes of the IP header. (Note that the 8 bytes immediately preceding the Payload are an 802.2 SNAP header.) When the GSN Header micropacket is carrying an ST transfer (EtherType=0x8181), the payload bytes in the Header micropacket are the initial 8 bytes of the ST Header.

Every HIPPI element is capable of processing GSN (HIPPI–6400-SC) Admin micropackets. These micropackets configure elements, discover the fabric topology, and maintain the elements of a GSN fabric. The TYPE field of the control bits (illustrated in Figure 1-3) indicates that a micropacket is of the Admin type. Admin micropackets have the format illustrated in Figure 1-4.

Most HIPPI-6400 elements have two ports: one leading toward the fabric and the other leading toward the host/core. For example, a link end element (such as the SuMAC ASIC) has one port connected to a physical link/the fabric and the other port connected to additional GSN logic (which may be another local element) on an adapter board. Notice that a GSN system may contain more than one element; this fact is important in understanding the processing of Admin micropackets.

An Admin micropacket can enter an element through either port, as illustrated in Figure 1-7. Each Admin micropacket is either processed and responded to or forwarded to the next element through the element's other port, as illustrated in Figure 1-7. A response to an Admin micropacket always exits the element through the same port by which the original Admin micropacket arrived.

Figure 1-7. Dual-port HIPPI–6400-PH Elements

The hop count field in the Admin micropacket determines when the Admin packet is acted upon/processed. The count indicates the number of elements (hops) through which the Admin micropacket is propagated/forwarded before it is processed. As long as the hop count is greater than zero, the receiving element decrements the hop count by one and transmits the Admin micropacket out the element's other port (which leads to another element), as illustrated in Figure 1-8. When the count is zero, the receiving element processes the micropacket and responds, as illustrated in Figure 1-9. Figure 1-10 through Figure 1-12 show examples of various hop count values and the manner in which hop count determines which element acts on and responds to the micropacket.

Table 1-6 lists the administrative commands that are available with Admin micropackets.

Figure 1-8. Hop Count >0 Indicates Forward Admin Micropacket

Figure 1-9. Hop Count =0 Indicates Process Admin Micropacket

Figure 1-10. Hop Count Example: hop_count = 0

Figure 1-11. Hop Count Example: hop_count = 1

Figure 1-12. Hop Count Example: hop_count = 2

The most salient feature of ST is that it prepares both endpoints for the data movement before any data is transmitted. The first step in the preparation is to create a condition (state) called a virtual connection or VC (described in “ST Connection Setup Sequence”). The second step is a handshake that allocates memory for the data movement and exposes this memory to the other endpoint (described in “ST Data Movement Sequences Including Memory Allocation”). There are two kinds of the memory–allocation handshake: one provides memory that is used once (described in “Single-use Memory Data Movements”); the other provides memory that is used many times until released (described in “Persistent Memory Data Movements”). The two endpoints exchange ST control operations to accomplish these prearrangements. Only after these prearrangements are complete can the first data movement begin; the data movement is performed with ST data operations.

The following terms have specific meanings within the context of ST:

The Operation is the basic protocol data unit for ST. Each ST Operation is carried within a single GSN Message, composed of two or more HIPPI–6400 micropackets, as illustrated in Figure 1-13.

Figure 1-13. ST Operation

ST operations (listed in Table 1-7) are commonly grouped into the following categories:

The ST Header (illustrated in Figure 1-14) carries the information that implements the ST protocol features. Some of the parameters that are communicated within the ST Header are:

The following are some of the endpoint behaviors that can be controlled by the operation's ST Header:

ST defines sequences of operations for accomplishing various tasks, including the following:

Each ST sequence allows the two endpoints to exchange a set of control parameters and information. The parameters are carried in the ST Header (illustrated in Figure 1-14). Each type of operation uses the Header fields differently and exchanges a different set of parameters.

Before any ST data can be exchanged, a Virtual Connection (VC) must be set up between the initiator and the responder. Upon successful completion of this exchange, each endpoint will have stored a set of parameters associated with the VC and will have set aside some resources for exclusive use by this VC. Three of the stored parameters are used (as a tuplet) for identifying/validating operations that arrive to the VC. The verification tuplet consists of: the remote endpoint's ST port number, the local endpoint's ST port number, and the key value that the local endpoint has assigned to this VC. Figure 1-15 illustrates how these identification parameters are set up.

---

Note: The initiator for the connection setup sequence is the endpoint that sends the first control operation for the sequence (that is, the Request_Connection).

---

The connection setup sequence consists of two control operations: a Request_Connection sent by the initiator, followed by a Connection_Answer sent by the responder). Figure 1-15 and Figure 1-16 illustrate different subsets of the information exchanged in one successful connection setup sequence. Figure 1-17 illustrates a connection setup sequence in which the responder refuses to create the VC.

The ST connection setup sequence negotiates and sets the following parameters and resources that remain in effect for the duration of the VC:

When an endpoint no longer wants a VC, it initiates the connection teardown sequence illustrated in Figure 1-18. This sequence is not used to terminate data movements. (See “ST Termination Sequence for a Data Movement”.)

Figure 1-18. ST Connection Teardown Sequence

This section describes ST data movement sequences. Each ST data movement sends upper-layer (user) data from one endpoint (the source) to one other endpoint (that is, one final destination). The entire data transfer is controlled by the VC parameters negotiated during one ST connection setup procedure (described in “ST Connection Setup Sequence”) or renegotiated during the data movement. The setup sequence must be completed before any data movement sequence is initiated.

The data movement sequences consist of two to five operations, exchanged between the VC's two endpoints (the memory-allocation handshake), followed by one or more data operations. There are five different data movement sequences, as summarized in Table 1-8. The initiator controls which sequence is used, depending on the type of memory it wants to have allocated, the type of functionality it desires for the data movement, and the role it wants to assume in the transfer.

The memory-allocation handshakes allow either of the following types of memory to be allocated for receipt of the data:

Table 1-8 summarizes the five data movement sequences and indicates where each sequence is illustrated:  

---

Note: Within a data movement sequence, the initiator is the endpoint that sends the first control operation for the sequence (for example, Request_to_Send or Request_Memory_Region), regardless of whether it operates as the data transmitter (source) or receiver (destination).

---

Table 1-9 summarizes the data size ranges for each type of data movement. As illustrated in Figure 1-19, the data is first chunked into one or more Blocks; the maximum size for a Block is negotiated during the memory allocation handshake. Each Block is divided into one or more scheduled transfer units (STU; the data for one data operation); the maximum size for the STU was negotiated during the connection setup sequence. Any ST data movement that is larger than the VC's maximum STU size requires multiple data operations. Each STU (that is, each data operation) is transmitted as one GSN Message. The flow-control mechanism for user data (described in “ST Flow Control Sequences”) operates at the Block level. 

Figure 1-19. Data Handling for ST Data Movements

Persistent Memory Data Movements

The persistent memory sequences consist of a few control operations (the memory–allocation handshake) followed by any number of Put, Get, and/or FetchOp sequences. The persistent memory handshake allocates one or more memory regions at the responding endpoint. These regions are then used multiple times; each buffer within each region is used over and over during the life of the virtual connection. When properly used, this method provides permanent, low–latency delivery, in which an unlimited number of transfers can be performed with no intervening overhead. There is an important caveat: the low latency on this type of data transfer depends on the speed at which the memory can be made available for the next use. This type of transfer works best for small (or fixed-size) data and for applications for which the transmission rate is well understood, so that the memory can be sized in a manner that allows it to be recycled within an acceptable period of time. It is the responsibility of the upper-layer applications to manage flow control and prevent precipitous overwriting of the memory region.

Once a persistent memory region has been allocated at the responder endpoint, the initiator can move data in or out of it in three manners:

• Put sequence (illustrated in Figure 1-20) 
  One data operation (STU) that writes any portion of or the entire persistent memory region at the responder. This sequence can be repeated over and over with no intervening operations.

• Get sequence (illustrated in Figure 1-21) 
  A GET control operation to expose memory at the initiator for receiving the requested data, followed by any number of data operations. Each data operation moves a portion or all of the data from the responder's allocated memory into the initiator's memory. Multiple GETs can be outstanding (occurring) simultaneously to different or shared portions of the persistent memory region.

• FetchOp sequence (illustrated in Figure 1-22 and Figure 1-23) 
  A FETCHOP control operation to expose memory at the initiator for receiving the retrieved data and to specify the desired function (increment, decrement, or clear). Then, a single data operation (one STU) that moves one 64-bit Block of data from the responder's memory into the initiator's memory. When the data arrives successfully at the initiator, the initiator issues a completion control message, at which point the responder performs the specified function on its own copy of the data. If the completion does not arrive within a timeout period, the responder retransmits the data. Note that, unlike PUT and GET, this data movement sequence is atomic.

A persistent memory region is terminated (released) with an End operation, as described in “ST Termination Sequence for a Data Movement”.

Figure 1-20. ST Data Movement Sequence: Persistent Memory—Put

Figure 1-21. ST Data Movement Sequence: Persistent Memory—Get

Figure 1-22. ST Data Movement Sequence: Persistent Memory—FetchOp

Figure 1-23. Example of FetchOp

Single-use Memory Data Movements

The single–use memory movement sequence consists of a few control operations (the memory-allocation handshake) that allocate memory at the destination endpoint, followed by one or more data operations for a specified amount of data. The data transfer uses the destination's allocated memory once; each buffer is used only once during the life of the transfer. This method allows high-bandwidth delivery after an initial delay for the allocation of resources: the transfer provides for a limited number of back–to–back writes or reads with no intervening overhead. This method is efficient for large, variable-length data.

A data transfer can be aborted (terminated before all the data has been transferred) with an End operation, as described in “ST Termination Sequence for a Data Movement”.

Figure 1-24 illustrates the data transfer sequence used when the initiator is the data source. Figure 1-25 illustrates the sequence used when the initiator is the data destination. Each illustration includes the memory allocation handshake.

Figure 1-24. ST Data Movement Sequence: Single-use Memory with Initiator as Source

Figure 1-25. ST Data Movement Sequence: Single-use Memory with Initiator as Destination

Flow control operates differently for data transfers and ST operations. Each is explained below.

Data Transfer Flow Control 

ST endpoints implement strict flow control for all data transfers done to single-use memory. For this purpose they use the Request_To_Send (RTS) and Clear_To_Send (CTS) control operations. There can be multiple CTSs generated in response to one RTS, as explained below and summarized in Table 1-10.

The ST flow control sequence regulates both the number of data transfer events that occur between the two endpoints and the size of these events. Before any data is transferred, the data transmitter (source) generates an RTS, in which it specifies the maximum size block of data and the number of blocks that it wants to send right now. The specified (requested) size and number do not oblige the receiver to give permission for that size or number; these are only suggestions that, if followed, could make the transfer more efficient.

The data receiver (destination) generates one or more CTSs in response to each RTS. In each CTS, the receiver gives the source permission to transmit one block of data; the number of CTSs issued by the receiver cannot exceed the number of “requested blocks” specified in the RTS. In the first CTS for the data movement, the receiver indicates the block size that it is willing to receive during this data movement; the block size must be no larger than the maximum block size specified in the associated RTS. Before issuing each CTS, the receiver must allocate the amount of memory specified by the block size in that CTS. See Figure 1-24 and Figure 1-25 for illustrations of the flow control sequence.

---

Note: ST does not use flow control for persistent memory data movements: Put, Get, and FetchOp.

---

Table 1-10. ST Flow Control Sequence

Transfer Event Parameter	Source Specifies	Destination Specifies
Number of events	In RTS: number of blocks the source would like to send at this time. Limitations: none.	In CTS: with each CTS, the destination gives the source permission to transmit 1 block of data. Limitations: Destination must not issue more CTSs than source has requested in its RTS. Destination must allocate memory for each CTS it generates.
Size of each event	In RTS: requested maximum block size for transfer events associated with this RTS. Limitations: none. Note: When the source does the actual data transfer, the size is not controlled by the RTS maximum block size; it is limited by the block size specified in the CTS.	In first CTS: block size that will be used for these transfer events. Limitations: Block size must not exceed maximum size specified in the associated RTS.

Operation Flow Control 

Flow control for the ST Headers of ST operations is managed with a mechanism called slot allocation. Each slot represents memory that has been allocated at an endpoint to hold one incoming ST Header while it awaits processing. All incoming ST Headers use one slot, except Request_Connection operations and Data operations that have the Silent flag set.

---

Note: Data operations with the Silent flag set, do not occupy a slot because the ST Header for these operations is not passed to the receiving endpoint (and hence is not stored). The Request_Connection operation does not occupy a slot because the VC does not yet exist when this operation arrives. An implementation may have a queue of slots associated with Port 0 (the port to which the Request_Connection arrives), but the queue is not required because there are no consequences caused by the endpoint dropping the request other than the initiator trying again, until it succeeds.

---

During the setup sequence for a VC, each endpoint communicates to the other endpoint the number of slots it has allocated for that VC. Updates for slot availability are communicated during normal operation with Request_State_Response operations. (See “ST Status Sequences” for details.) Each source keeps track of the number of outstanding operations (that is, slot-consuming ST Headers that it sends) and makes sure that it does not send more operations than the destination can handle.

During normal operation, the endpoints for a VC can use either of two status sequences (illustrated in Figure 1-26 and Figure 1-27) to obtain information from the other endpoint about its state and status.

Figure 1-26. Status Sequence Using Request_State

Figure 1-27. Status Sequence Using S Flag in ST Header

The information that can be exchanged with this mechanism includes:

The following data movements do not have a natural ending:

To terminate either of the above data movements and release the associated resources, either endpoint initiates the termination sequence illustrated in Figure 1-28. In addition, this sequence can be used to abort a data transfer of specific length before all the data has been transferred.

Figure 1-28. Termination Sequence

The hosts connected to a GSN or HIPPI fabric do not have to function as one network address space. The hosts can be organized into smaller groupings (for example, based on function, project, or hardware manufacturer). Each grouping of hosts is a separate logical network or a LIS. Each LIS is assigned a sequence of network-layer addresses (that is, a unique address space). Figure 1-31 illustrates this concept.

A group's address space can be the complete range of addresses for an INET network address (192.0.2.0 to 192.0.2.255), or it can be a portion of the range (for example, subnet 192.0.2.0 to 192.0.2.31). Membership in a group is determined for each GSN network interface (for example, each gsn#) by the INET address associated with the interface (in the /etc/config/netif.options file) and the netmask value (in the ifconfig.options or the ifconfig-#.options file). The netmask value defines the size of the address space for each group. For example, a netmask value of 0xFFFFFF00 creates an address range that provides 256 individual host addresses. However, netmask value 0xFFFFFFE0 (shown in Figure 1-30) creates eight LISs in which each LIS can have up to 32 “host” addresses.

A logical network or a LIS can include hosts from physically different GSN and HIPPI fabrics, as long as there is a “bridging” communication path between the fabrics. Hosts that are members of the same INET address space (thus benefitting from the services provided by broadcast and routing) do not have to be physically attached to the same physical medium (fabric). Figure 1-32 illustrates this concept.

Direct communication between INET hosts (without use of an intermediate router) can occur only when the network interfaces involved in the exchange are members of the same logical address space (network or LIS). Contact with members outside one's own LIS requires use of an INET address router.

This rule is true even when a shared hardware connection (for example, a switch) exists between the two hosts that belong to different LISs. For example, for two hosts attached to the same switch, a message from host A in LIS 1, if sent to host B in LIS 2, must go through host C, an INET router. The benefit is that, no matter where a GSN network adapter is physically located or relocated, it continues to function as a member of the same LIS. Notice that no address or LIS–membership change is required when an endpoint is physically relocated.

The following facts explain why this concept exists:

The basic concepts summarized in “GSN Fabrics and Logical Networks”, make the examples described in this section possible.

Figure 1-31 and Figure 1-32 show examples of subnetting within two different GSN fabric configurations. The LIS addressing used in these examples (summarized in Figure 1-30) is identical. The examples use network INET address 192.0.2, so that each host address is 192.0.2.xxx. Hosts in LIS_1 use addresses between 192.0.2.0 and 192.0.2.31; those in LIS_2 use addresses between 192.0.2.32 and 192.0.2.63, and so on.

Figure 1-30. Subnet Mask for Examples

If you want a single-fabric site to have multiple address spaces, you can use multiple INET network addresses, or you can use a netmask to divide a single INET address space into smaller chunks (referred to as LISs). Likewise, in a multiple–fabric site, you can group all the hosts into one logical address space, or into multiple LISs regardless of each host's location.

Figure 1-31 illustrates a GSN fabric that has one switch to which all the network interfaces are attached (that is, all endpoints in this fabric have a direct physical link to one another). The example shows two LISs. Communication from A in LIS_1 to C in LIS_2 passes through the router (network interfaces J and H). Messages do not go directly from endpoint A to C, because of the concept explained in “Basic Concept #3”.

Figure 1-31. Single-switch GSN Fabric with LISs

Figure 1-32 illustrates a different configuration for the same address space and network interfaces (“hosts”) used in Figure 1-31. This configuration is a two-switch fabric. In this example, A, B, E, J, K, and L belong to LIS_1, while C, D, F, G, and H belong to LIS_2. The system with network interfaces H and J continues to perform as the router between the two LISs. Just as in the first example (Figure 1-31), communication directed to C in LIS_2 from A in LIS_1, goes first to the router (J/H), even though both A and C are physically attached to the same switch. But, most importantly, notice that the router has been moved to a different switch, and yet, the INET addressing is identical to that used in the first configuration. The hardware changes do not affect the addressing. Also note that a router for an LIS does not need to share a switch with the members of its LISs, as illustrated by router J in relation to hosts A and B and router H in relation to hosts C and D.

Figure 1-32. Multiple-switch GSN Fabric with LISs

Figure 1-33. LIS Membership That Spans Fabrics

A GSN fabric is said to support broadcasting when all of the switches of that fabric provide broadcasting. The behavior for HARP clients and HARP servers is different depending on whether the underlying GSN/HIPPI fabric supports broadcasting.

The address resolution protocol for HIPPI networks is specified in the HARP RFCs. The protocol works with fabrics that provide broadcasting and with those that do not. One of the first tasks of each HARP client is to determine if its underlying fabric supports broadcasting, as described in “Determining Fabric Support for Broadcast”.

HARP provides a dynamic, client/server-based address resolution service. The protocol makes it possible for each IP/ST-over-HIPPI endpoint (client) within a network to register or communicate its own INET address and ULA, and to discover the ULAs for hosts with whom it wants to communicate. The HARP server maintains a kernel-resident lookup table that maps INET addresses to ULAs. HARP occurs in two phases: a registration phase (summarized in the section “HARP Registration Phase”) and a normal operation phase (summarized in “HARP Normal Operation Phase”).

When an LIS includes one or more endpoints that do not support dynamic HARP, static mappings for those endpoints must be added to the address resolution table at the HARP server (as described in the section “HARP Normal Operation Phase”).

HARP Registration Phase

During initialization of each GSN device, a HARP client on a non-broadcast medium will register its address pair (INET address and ULA) with each HARP server on its LIS. This is done by transmitting an InARP request to each HARP server. The InARP request contains the IP-to-ULA mapping of the client, and it requests the IP-to-ULA mapping of the server in reply. (Since InARP requests are sent to ULAs, each client must know the ULAs for all servers on the LIS. For IRIX, this information is contained in the configuration file /etc/config/harpd.options. For details, see “Edit harpd.options File” in Chapter 2.)

Because HARP clients can be brought up before HARP servers, a client might not receive replies to all (or any) of the InARP requests that it transmits. For each nonresponding HARP server, a HARP client will periodically retransmit the InARP request.

When at least one HARP server has responded with an InARP reply, the HARP client gains the ability to resolve unknown IP-to-ULA mappings on the LIS; the client then transitions from the registration phase to the operational phase.

Figure 1-34. HARP Registration

When a host within a HIPPI/GSN LIS does not support dynamic HARP, the system administrator needs to add a static entry for that host to each HARP client (for broadcast capable networks) or to each HARP server's database (for nonbroadcast-capable networks). Static entry definitions can be placed into the HARP daemon configuration file (/etc/config/harpd.options), or they can be added manually to the mapping database by using the gsnarp utility. Each entry in the database must map a ULA (IEEE or MAC address) to an INET address.

These guidelines explain how to select a system to provide HARP services (that is, be the HARP server) when the HIPPI fabric does not support broadcasting. It is not necessary to identify a system for this purpose when the fabric supports broadcasting.

From among the members of the LIS, at least one system must be chosen to be a HARP server. For redundancy purposes, at least two systems should be selected for this purpose. (When no HARP servers are available on an LIS, no HARP address resolution can occur, so the only members of the LIS who will be able to intercommunicate are hosts whose HARP entries are statically defined.

To ensure that every HARP server's database contains a complete mapping for all registered hosts, all hosts in an LIS must identify the same systems as HARP servers.

The IRIX GSN implementation of the ST protocol uses the same address resolution scheme as is used for IP–over–GSN. See “Address Resolution for GSN” for the details.

---

Note: Each gsn# network interface services two protocols: ST and IP. The INET address assigned to an instance of gsn# is shared by the ST-over-GSN and IP-over-GSN stacks. Some of the upper-layer address processing (for example, routing) that is performed on the address applies to both IP and ST traffic.

---

These entries are loaded when the harpd daemon is initialized via the harpd configuration file (by default, /etc/config/harpd.options; for details, see “Edit harpd.options File” in Chapter 2). Alternatively, the administrator can add them individually via the gsnarp -s command. The administrator can remove static entries via gsnarp -d.

IRIX HARP maintains the dynamic entries in its HARP table in conformance with the HARP standard. It adds entries as it learns about them, refreshes them as they are reregistered by their owners (the clients), and ages and deletes entries as they go stale.

With each restart (for example, a power on, a reboot or init 0 command), the startup routine probes for hardware on all the modules connected into the CrayLink interconnect fabric. All the slots and links in all the modules within the fabric are probed. The routine then creates a hierarchical filesystem, called the hardware graph, that lists all the located hardware. The top of the hardware graph is visible at /hw. For complete details, see the man page for hwgraph(4). After the hardware graph is completed, the ioconfig program assigns a unit number to each located device that needs a number. Other programs (for example, hinv and each device's driver) read this assigned number and use it.

The XIO slots are searched (probed for a device) in the order shown below; this order is not the same sequence as the XIO slot numbering. For example, the device in XIO slot 4 is located before the device in slot 2 and, because of this, may have a lower unit number than the device in slot 2. After the first power on, you can edit the /etc/ioconfig.conf file to assign unit numbers that are convenient for you. Your changes are used during each subsequent power on. See the ioconfig(1M) man page for further details.

On an initial system startup, ioconfig groups devices into classes/types and assigns hardware unit numbers sequentially within each class. It records these assignments in the /etc/ioconfig.conf file; for example, if two SGI GSN products are found, they are numbered unit 0 (gsn0) for the first one found and unit 1 (gsn1) for the second one. When an SGI GSN product is a two-board solution, both boards are associated with a single unit number. On subsequent startups, ioconfig distinguishes between hardware that it has seen before and new items. To previously seen items, it assigns the same hardware unit numbers (those that are recorded in the ioconfig.conf file). To new hardware, it assigns new sequential numbers and records them. It never reassigns a number, even if the device that had the number is removed and leaves a gap in the numbering. For example, in a system with two instances of some class of devices, if the unit0 is removed, the next restart results in the system listing only unit1; if a new board is installed in a new location, it is listed as unit2.

New items are differentiated from previously seen items through the hardware graph listing (that is, the path under /hw/module/#/slot/io#/...). The database of previously seen devices is kept in the file /etc/ioconfig.conf. A replacement board (with the exact same hardware device name) that is installed into the location of an old board (so that it has the same hardware graph listing) is assigned the old board's unit number, but a board that is moved from one location to another is assigned a new number. For example, in a two-device system with ioconfig.conf entries illustrated below, if unit0 is moved to a different slot, the next restart results in a new item in the ioconfig.conf file. The hinv command lists unit1 (an original board in its original slot) and unit2 (the board that has been moved to a new slot), but not unit0. For more information about the hardware graph and ioconfig, see the man pages for hwgraph(4) and ioconfig(1M).

Initial entries for two devices: 
0 /hw/module/1/slot/io8/xio_gsn/device
1 /hw/module/1/slot/io4/xio_gsn/device
0 /hw/gsn/0
1 /hw/gsn/1
Entries after unit0 is moved: 
0 /hw/module/1/slot/io8/xio_gsn/device
1 /hw/module/1/slot/io4/xio_gsn/device
2 /hw/module/1/slot/io5/xio_gsn/device
1 /hw/gsn/1
2 /hw/gsn/2

The two-board SGI GSN product occupies two XIO slots that are logically associated with a single device (one unit number). The device has two XIO slots and two hardware graph entries. All links (for example, the short or convenience path, /hw/gsn/#) point to the XIO slot for the main SGI GSN board. All located SGI GSN hardware devices can be displayed with the /sbin/hinv or find command.

As the startup process continues, it calls the network drivers and protocol software modules so that they can create their network and programmatic interfaces. For GSN, this step works in the following manner: