The cpuset facility is primarily a workload manager tool permitting a system administrator to restrict the number of processors and memory resources that a process or set of processes may use. A cpuset defines a list of CPUs and memory nodes. A process contained in a cpuset may only execute on the CPUs in that cpuset and may only allocate memory on the memory nodes in that cpuset. Essentially, cpusets provide you with a CPU and memory containers or "soft partitions" within which you can run sets of related tasks. Using cpusets on an SGI Altix system improves cache locality and memory access times and can substantially improve an applications performance and runtime repeatability. Restraining all other jobs from using any of the CPUs or memory resources assigned to a critical job minimizes interference from other jobs on the system. For example, Message Passing Interface (MPI) jobs frequently consist of a number of threads that communicate using message passing interfaces. All threads need to be executing at the same time. If a single thread loses a CPU, all threads stop making forward progress and spin at a barrier. Cpusets can eliminate the need for a gang scheduler.

Cpusets are represented in a hierarchical virtual file system. Cpusets can be nested and they have file-like permissions.

In addition to their traditional use to control the placement of jobs on the CPUs and memory nodes of a system, cpusets also provide a convenient mechanism to control the use of Hyper-Threading Technology.

For detailed information on cpusets, see Chapter 6, “Cpusets on SGI ProPack 4 for Linux” in the Linux Resource Administration Guide.

The port of Comprehensive System Accounting (CSA) software packages from IRIX to Linux is the result of an open source collaboration between SGI and Los Alamos National Laboratory (LANL) to provide jobs-based accounting of per-task resources and disk usage for specific login accounts on Linux systems.

Providing extensive system accounting capabilities is often important for very large systems, especially when the system will be shared or made available for other organizations to use. CSA uses a Job Containers feature, which provides on Linux the notion of a job. A job is an inescapable container and a collection of processes that enables CSA to track resources for any point of entry to a machine (for example, interactive login, cron job, remote login, batched workload, and so on).

The Linux CSA application interface library allows software applications to manipulate and obtain status about Linux CSA accounting methods.

CSA on Linux is an SGI open source project, also available from the following location:

http://oss.sgi.com/projects/csa

For further documentation and details on CSA support, see the chapter titled “Comprehensive System Accounting” in the Linux Resource Administration Guide.

SGI provides the ability to divide a single SGI Altix system into a collection of smaller system partitions. Each partition runs its own copy of the operating system kernel and has its own system console, root filesystem, IP network address, and physical memory. All partitions in the system are connected via the SGI high-performance NUMAlink interconnect, just as they are when the system is not partitioned. Thus, a partitioned system can also be viewed as a cluster of nodes connected via NUMAlink.

Benefits of partitioning include fault containment and the ability to use the NUMAlink interconnect and global shared memory features of the SGI Altix to provide high-performance clusters.

For further documentation and details on partitioning, see the Linux Configuration and Operations Guide.

Persistent PCI bus numbering ensures that bus numbers can remain the same across reboots in case of faulty hardware or reconfiguration. During platform initialization, as buses are discovered, they are assigned a logical bus number. Each logical bus number is unique, systemwide. The default number of buses supported by SGI Altix 3000 systems is 256 (numbered 0 to 255).

By default and for IX-bricks and PX-bricks only, bus numbers are allocated starting from the lowest C-brick module ID to which the I/O brick is connected. For the NUMAlink connected PA-bricks it is the lowest module ID of the PA-brick. Since you can have both IX-bricks and PX-bricks on a system along with PA-bricks, it is the lowest of all these combinations.

An I/O brick can be any one of the following:

Each I/O brick is allocated 0x10 buses, although the current I/O bricks support only six buses each. Therefore, bus numbers are not contiguous across the system. Bus numbers are sparse and have holes in them.

On an SGI Altix 3000 series systems, the IX-bricks and PX-brick I/O bricks have six physical buses and a PA-brick I/O brick has four physical buses. These buses are numbered 0x1 through 0x6, left to right, looking at the back of the I/O brick. If there is more than one I/O brick on the system, the buses on the next I/O brick are numbered 0x11 through 0x16. The rationale for this numbering is that the bus numbers of each I/O brick are stamped on the back of the brick and they start from 1. Therefore, the right most digit of a bus number corresponds to the actual stamped number on the I/O brick An SGI Altix 350 system has 2 buses per module.

If you have only one I/O brick, you do not need persistent bus numbering. However, if you have more than one I/O brick, persistent bus numbering is strongly recommended, so that if an I/O brick fails to boot, your bus numbers are still the same.

To make use of persistent bus numbering, you must set the PciBusList1, PciBusList2, and PciBusList3 PROM environment variables. Each variable is a comma separated list of the moduleid of I/O Bricks representing the order you want the persistent bus numbering.

There are three PciBusList# variables to support persistent bus numbering for 16 I/O bricks. (This is because PROM variables are limited to 48 characters. That gives us six moduleid per variable to support up to 16 I/O bricks). Some examples are, as follows:

POD> setallenv PciBusList1 "001i01,001p04,101p10"

or

POD> setallenv PciBusList1 "001i01,001p04,101p10,010i10,011p12,012p11"
POD> setallenv PciBusList2 "102i01"

The PROM clearance PciBusList1 variable is used to clear the persistent bus numbering.

An Ethernet interface can be given a persistent internet addresses by associating its permanent MAC address, such as 08:00:69:13:f1:aa, with an internet protocol (IP) address, for example 192.168.20.1. An interface with a persistent IP address will be given the same IP address each time the system is booted.

SGI ProPack 4 is built on top of SLES9. SLES9 has its own implementation of Ethernet persistent naming, that is, the Media Access Control (MAC) address is embedded in the name of the configuration file of the associated Ethernet interface.

For example, if you want to ensure that the Ethernet interface whose MAC address is 08:00:69:13:f1:aa always uses the IP address 192.168.20.1, edit the /etc/sysconfig/network/ifcfg-eth-id-08:00:69:13:f1:aa file so that it contains the following:

BOOTPROTO="static"
STARTMODE="onboot"
IPADDR="192.168.20.1"
NETMASK="255.255.255.0"
NETWORK="192.168.20.0"
BROADCAST="192.168.20.255"

For more information on Ethernet device configuration files and instructions for editing them, see /usr/share/doc/packages/sysconfig/README.

On an SGI Altix 4700 system, a PCI domain is a functional entity that includes a root bridge, subordinate buses under the root bridge, and the peripheral devices it controls. Separation, management, and protection of PCI domains is implemented and controlled by system software.

Previously, a PCI device was identified by bus:slot:function. With new I/O blade technology, a PCI device is identified by domain:bus:slot:function.

Domains (sometimes referred to as a PCI segment) are numbered from (0 to ffff). bus (0 to ff), slot (0 to 1f) and function (0 to 7).

On an SGI Altix 4700 system, a domain is a root bridge (with the bus being numbered zero), subordinate buses under that root bridge are buses 1-255.

In the past, a PA-brick was numbered bus 01, 02, 03, 04 (A bus for each root bridge (that is, each TIO ASIC)).

With PCI domain support each root bridge is numbered with a domain/bus number 0001:00, 0002:00, 0003:00,0004:00. If a subordinate bus is plugged into the root bridge bus, it has the same domain number as the root bridge, but a different bus number (for example, 0001:01).

For PCI Express the PCIE root complex is its own domain. Each port is a subordinate bus under that domain.

Domain numbers are allocated starting from the lowest module ID (that is, rack/slot/blade location).

The SGI XFS filesystem provides a high-performance filesystem for Linux. XFS is an open-source, fast recovery, journaling filesystem that provides direct I/O support, space preallocation, access control lists, quotas, and other commercial file system features. Although other filesystems are available on Linux, performance tuning and improvements leveraged from IRIX make XFS particularly well suited for large data and I/O workloads commonly found in HPC environments.

For more information on the XFS filesystem, see XFS for Linux Administration.

The SGI XVM Volume Manager provides a logical organization to disk storage that enables an administrator to combine underlying physical disk storage into a single logical unit, known as a logical volume. Logical volumes behave like standard disk partitions and can be used as arguments anywhere a partition can be specified.

A logical volume allows a filesystem or raw device to be larger than the size of a physical disk. Using logical volumes can also increase disk I/O performance because a volume can be striped across more than one disk. Logical volumes can also be used to mirror data on different disks.

This release adds a new XVM multi-host failover feature. For more information on this new feature and XVM Volume Manager in general, see XVM Volume Manager Administrator's Guide.

The SGI Message Passing Toolkit (MPT) provides industry-standard message passing libraries optimized for SGI computers. On Linux, MPT contains MPI and SHMEM APIs, which transparently utilize and exploit the low-level capabilities within SGI hardware, such as memory mapping within and between partitions for fast memory-to-memory transfers and the hardware memory controller's fetch operation (fetchop) support. Fetchops and other shared memory techniques enable ultra fast communication and synchronization between MPI processes in a parallel application.

MPI jobs can be launched, monitored, and controlled across a cluster or partitioned system using the SGI Array Services software. Array Services provides the notion of an array session, which is a set of processes that can be running on different cluster nodes or system partitions. Array Services is implemented using Process Aggregates (PAGGs), which is a kernel module that provides process containers. PAGGs has been open-sourced by SGI for Linux.

For more information on the Message Passing Toolkit, see the Message Passing Toolkit (MPT) User's Guide.

The SGI Performance Co-Pilot software was ported from IRIX to Linux to provide a collection of performance monitoring and performance management services targeted at large, complex systems. Integrated with the low-level performance hardware counters and with MPT, Performance Co-Pilot provides such services as CPU, I/O, and networking statistics; visualization tools; and monitoring tools.

For more information on Performance Co-Pilot, see the Performance Co-Pilot for IA-64 Linux User's and Administrator's Guide.

Programmable read-only memory (PROM) chips are placed in your computer at the factory with software programmed into them that allows the CPU to boot and allows you to perform system administration and software installations. The PROM chips are not part of your disk or your operating system; they are the lowest level of access available for your system. You cannot erase them or bypass them. For more information on PROM and the L1 controller, see the SGI L1 and L2 Controller Software User's Guide.

The Extensible Firmware Interface (EFI), a supporting platform to provide input to the CPU and to handle its output, is provided by SLES9 SP3, the base Linux operating system for SGI Altix systems running SGI ProPack 4 SP3. EFI also controls the server's boot configuration, maintaining the boot menu in durable, novolatile memory.

SLES9 SP3 uses elilo package which places the bootloader (elilo.efi) and configuration file (elilo.conf) in the /boot/efi/efi/SuSE/ directory on SGI Altix systems.

---

Note: When booting from SLES9 CDs, use the bootia64 command instead of elilo. Once the system is running SLES9 SP3, use elilo to boot from EFI.

---

---

Note: If you have installed multiple kernel images and want to boot with one that is not currently the system default (vmlinuz in /boot/efi/efi/SuSE), simply copy the vmlinuz and initrd files for the kernel you wish to use from /boot to /boot/efi/efi/SuSE.

---

For a summary of EFI commands, see Table 3-2.

FLEXlm is a flexible license management system from Macrovision that lets independent software vendors (ISVs) license their products and helps system administrators install and manage licenses with minimal overhead. It supports a wide range of licensing options, including simple node-locked licenses and floating licenses with redundant servers.

To build licensed software, ISVs must purchase a set of keys from Macrovision. System administrators can install license servers anywhere. Products purchased from SGI are typically licensed using FLEXlm.

For more information, visit

http://www.macrovision.com/solutions/esd/flexlm/flexlm.shtml

SGIconsole is a combination of hardware and software that provides console management and allows monitoring of multiple SGI servers running the IRIX operating system and SGI ProPack for Linux. These servers include SGI partitioned systems and large, single-system-image servers, including SGI Altix 350 systems and the SGI Altix 3000 family of servers and superclusters.

SGIconsole consists of an 1U rackmountable SGI server based on the Intel Pentium processor, a serial multiplexer or Ethernet hub, and a software suite that includes the Console Manager package and Performance Co-Pilot, which provides access to common remote management tools for hardware and software.

Console Manager is a graphical user interface for the SGIconsole management and monitoring tool used to control multiple SGI servers. SGIconsole also has a command line interface. For more information on SGIconsole, see the SGIconsole Start Here.

This section describes the system controller firmware used in SGI Altix systems.

The L1 and L2 controllers are system controller firmware used in SGI systems.

The L1 controller is embedded in each brick in SGI Origin and Onyx 3000 series systems and in SGI Altix 3000 systems. An SGI Altix 4700 system can have multiple L1 host connections depending on the size of the system. The L1 controller provides power and control sequencing, along with temperature and power monitoring for each brick.

The L2 controller is a rack-level controller that monitors and controls the bricks in its rack. All L2 controllers in a system are networked together and they consolidate the control and monitoring information from each brick to provide system-level control and monitoring.

The following references describe how to install and connect to an L1 or L2 controller:

For more information on the L1 and L2 system controller firmware, see the SGI L1 and L2 Controller Software User's Guide.

The L3 controller is a Linux software package that resides on a PC workstation or laptop computer. It provides remote support, system controller consoles, and system-level maintenance functions. The L3 controller software is obtained through the "System Controller Software 1.x" release.

The dlook(1) command displays the memory map and CPU use for a specified process. The following information is printed for each page in the virtual address space of the process:

Optionally, the amount of elapsed CPU time that the process has executed on each physical CPU in the system is also printed.

The dplace(1)  command binds a related set of processes to specific CPUs or nodes to prevent process migrations. In some cases, this tool improves performance because of the occurrence of a higher percentage of memory accesses to the local node.

For more information on NUMA tools, see Chapter 5, “Data Placement Tools” in the Linux Application Tuning Guide.

OpenGL Multipipe enables owners of high performance visualization solutions from Silicon Graphics to create high-resolution SGI Reality Center environments and achieve higher levels of performance while continuing to use existing single pipe applications. It brings the power of scalable visualization to new applications and enables organizations to create better products, generate new ideas, and increase operational efficiency.

OpenGL Multipipe enables high levels of screen space scaling and low-to-moderate levels of performance scaling without application modification. OpenGL Multipipe uses a flexible architecture that enables a logical screen to be built up from from individual physical screens that can be located anywhere within the logical screen space. This enables the creation of large walls with 2D arrays of displays, wide panoramic screens with overlapping displays for edge blending and the use of a scalable graphics compositor to drive a single screen with increased performance. For more information about OpenGL Multipipe, visit http://www.sgi.com/products/software/multipipe/ .

OpenGL Multipipe SDK is an API layer for OpenGL that provides a simplified approach to managing graphics applications across multiple graphics subsystems or pipes and focused on delivering run-time scalability on SGI scalable graphics servers.

Applications written using the OpenGL Multipipe SDK API can run seamlessly from desktop single-processor, single-pipe systems up to large multiprocessor, multipipe scalable graphics systems. A simple configuration file defines the desired configuration of the application, such that the same application can support normal office environments, SGI Reality Center rooms, walls, immersive desk displays, and domes. In addition, the same application can take advantage of scalable graphics systems by combining the power of multiple graphics pipes into a single display output, which in turn can be scaled to drive multiple display devices. OpenGL Multipipe SDK provides a straightforward solution for graphics applications that were designed for single-processor, single-pipe applications to scale seamlessly when run on graphics systems with multiple processors and multiple graphics pipes. For more information about OpenGL Multipipe SDK, visit http://www.sgi.com/products/software/multipipe/sdk/ .

OpenGL Performer is a powerful and comprehensive programming interface for developers creating real-time visual simulation and other professional performance-oriented 3D graphics applications. The toolkit simplifies development of applications used for visual simulation, manufacturing, simulation-based design, virtual reality, scientific visualization, interactive entertainment, broadcast video, architectural walk-through, and computer-aided design.

The latest release is built atop the industry-standard OpenGL graphics library, interoperates with OpenGL Volumizer, OpenGL Multipipe SDK, and OpenGL Vizserver; includes both ANSI C and C++ bindings; and is available for the IRIX, Linux, Windows XP, and Windows 2000 operating systems. For more information about OpenGL Performer, visit http://www.sgi.com/software/performer/.

OpenGL Vizserver is a technical and creative computing solution designed to deliver superior visualization, collaboration, and advanced performance to any client, whether on a desktop workstation or wireless tablet. OpenGL Vizserver allows users to remotely view and interact with large data sets from any other system at any location in an organization and to collaborate with multiple colleagues using these same applications and data.

OpenGL Vizserver enables a single SGI advanced visualization system to act as a multiuser graphics server which distributes remote and collaborative visualization sessions to client systems running a mix of desktop operating systems. With OpenGL Vizserver, graphics processing is handled entirely on the SGI advanced visualization system using full 3D hardware acceleration; the client machine simply decompresses and displays the image. This leverages existing internal infrastructure and allows customers to leverage the power of the world-class SGI advanced visualization systems on common desktop workstations such as Silicon Graphics visual workstations, Apple Macintosh workstations, Sun Solaris systems, or Intel architecture PCs running the Linux, Microsoft Windows NT, Windows 2000, or Windows XP operating systems. For more information about OpenGL Vizserver, visit http://www.sgi.com/products/software/vizserver/ .

OpenGL Volumizer is a cross-platform, high-level volume rendering API for the energy, manufacturing, medical, and sciences markets. OpenGL Volumizer is a revolutionary graphics API designed for interactive, high-quality, scalable visualization of large volumetric data sets. OpenGL Volumizer provides a high-level interface to OpenGL hardware to allow application writers and researchers to visualize multiple gigabytes of volumetric data. The API uses OpenGL for volume rendering and hence allows standard graphics applications to treat volumetric and surface data in a similar fashion.

OpenGL Volumizer is a library of C++ classes that facilitates the manipulation and display of volumetric data sets common in geo-science, medical, scientific and engineering applications. It provides a layer of functionality that sits on top of OpenGL and integrates seamlessly into higher-level toolkits and applications. For more information on OpenGL Volumizer, visit http://www.sgi.com/products/software/volumizer/ .

The digital media libraries are a family of libraries that provide application program interfaces (APIs) for digital media I/O, file operations, playback, and conversions. For more information, see the Digital Media Programming Guide.

The digital media tools are a suite of end user tools that enable you to capture, edit, record, play, compress, and convert audio, video, or image files. For more information, see the Digital Media Tools Guide (007-2275-012). You can find this manual on a system with digital media packages installed or on the SGI Technical Publications Library.