CpuMemSets provides a Linux kernel facility that enables system services and applications to specify on which CPUs they can be scheduled, and from which nodes they can allocate memory. The SGI ProPack kernel and library installation automatically include support for CpuMemSets.

The default configuration makes all CPUs and all system memory available to all applications. You can use the CpuMemSets facility to restrict any process, process family, or process virtual memory region to a specified subset of the system CPUs and memory.

The runon command relies on CpuMemSets to enable a user to run a specified command on a specified list of CPUs. Both a C shared library and Python language module are provided to access the CpuMemSets system interface.

In future releases of SGI ProPack for Linux, SGI anticipates providing additional facilities that provide other system services that provide convenient access to the CpuMemSets facility.

For further documentation and details on CpuMemSets support, see the chapter titled “CPU Memory Sets and Scheduling” in the Linux Resource Administration Guide or browse the files that are installed as part of the CpuMemSets RPM, as listed by the following command:

rpm -ql CpuMemSets

In particular, see the man pages for cpumemsets(2) and runon(8).

CpuMemSets is an SGI open source project, also available from the following location:

http://oss.sgi.com/projects/cpumemsets

The cpuset system is primarily a workload manager tool that permits a system administrator to restrict the number of processors that a process or set of processes can use.

A cpuset is a named set of CPUs, which can be defined to be restricted or open. A restricted cpuset allows only processes that are members of the cpuset to run on the set of CPUs. An open cpuset allows any process to run on its CPUs, but a process that is a member of the cpuset can run only on the CPUs belonging to the cpuset. A cpuset is defined by a cpuset configuration file and a name.

A system administrator can use cpusets to create a division of CPUs within a larger system. Such a divided system allows a set of processes to be contained to specific CPUs, reducing the amount of interaction and contention those processes have with other work on the system. In the case of a restricted cpuset, the processes that are attached to that cpuset will not be affected by other work on the system; only those processes attached to the cpuset can be scheduled to run on the CPUs assigned to the cpuset. An open cpuset can be used to restrict processes to a set of CPUs so that the effect these processes have on the rest of the system is minimized. For further documentation and details on cpusets, see chapter 5, “Cpuset System” in the Linux Resource Administration Guide.

The port of Comprehensive System Accounting (CSA) software packages from IRIX to Linux is the result of an open source collaboration between SGI and Los Alamos National Laboratory (LANL) to provide jobs-based accounting of per-task resources and disk usage for specific login accounts on Linux systems.

Providing extensive system accounting capabilities is often important for very large systems, especially when the system will be shared or made available for other organizations to use. CSA uses a Job Containers feature, which provides on Linux the notion of a job. A job is an inescapable container and a collection of processes that enables CSA to track resources for any point of entry to a machine (for example, interactive login, cron job, remote login, batched workload, and so on).

The Linux CSA application interface library allows software applications to manipulate and obtain status about Linux CSA accounting methods.

CSA on Linux is an SGI open source project, also available from the following location:

http://oss.sgi.com/projects/csa

For further documentation and details on CSA support, see the chapter titled “Comprehensive System Accounting” in the Linux Resource Administration Guide.

SGI provides the ability to divide a single SGI Altix 3000 system into a collection of smaller system partitions. Each partition runs its own copy of the operating system kernel and has its own system console, root filesystem, IP network address, and physical memory. All partitions in the system are connected via the SGI high-performance NUMAlink interconnect, just as they are when the system is not partitioned. Thus, a partitioned system can also be viewed as a cluster of nodes connected via NUMAlink.

Benefits of partitioning include fault containment and the ability to use the NUMAlink interconnect and global shared memory features of the Altix 3000 to provide high-performance clusters.

For further documentation and details on partitioning, see the Linux Configuration and Operations Guide.

Persistent PCI-X bus numbering ensures that bus numbers can remain the same across reboots in case of faulty hardware or reconfiguration. During platform initialization, as buses are discovered, they are assigned a logical bus number. Each logical bus number is unique, systemwide. The default number of buses supported by SGI Altix 3000 systems is 256 (numbered 0 to 255).

By default, bus numbers are allocated starting from the lowest C-brick module ID to which the I/O brick is connected. An I/O brick is either an IX-brick or a PX-brick on SGI Altix 3000 systems. Each I/O brick is allocated 0x10 buses, although the current I/O bricks support only six buses each. Therefore, bus numbers are not contiguous across the system. Bus numbers are sparse and have holes in them.

An I/O brick has six physical buses. These buses are numbered 0x1 through 0x6, left to right, looking at the back of the I/O brick. If there is more than one I/O brick on the system, the buses on the next I/O brick are numbered 0x11 through 0x16. The rationale for this numbering is that the bus numbers of each I/O brick are stamped on the back of the brick and they start from 1. Therefore, the rightmost digit of a bus number corresponds to the actual stamped number on the I/O brick.

If you have only one I/O brick, you do not need persistent bus numbering. However, if you have more than one I/O brick, persistent bus numbering is strongly recommended, so that if an I/O brick fails to boot, your bus numbers are still the same.

To make use of persistent bus numbering, you must supply the ioconfig= parameter to the kernel.

The ioconfig= parameter takes a comma-separated list of I/O brick numbers as an argument. The following manual boot example uses elilo. It tells the kernel that the I/O brick represented by 101.01 is assigned bus numbers 0x1 through 0x6 and the I/O brick represented by 101.02 is assigned bus numbers 0x11 through 0x16.

Shell> elilo vmlinux ioconfig="101.01,101.02" root=/dev/xscsi/pci01.03.0-1/target1/lun0/part3

You can find the numbers to use with ioconfig by looking at the components of your system from your L2 controller, as in the following example:

l2-pumpkin-001-L2>pwr
001c11:
power appears on
001c27:
power appears on
101i01:
power appears on
101p02:
power appears on

In this example, 101i01 is an IX-brick and represents the 101.01 in the previous elilo boot example. The 101p02 notation is a PX-brick and represents 101.02.

Most customers will have their system set up to boot automatically. This means that you should update your elilo.conf file with the ioconfig parameter. The . elilo.conf file is available in the /boot/efi/EFI/sgi directory on your Linux system. The following is an example elilo.conf file:

prompt
timeout=50
relocatable
default=sgilinux
append="ioconfig=101.01,101.02"

image=vmlinuz-2.4.19-sgi21r4
        label=sgilinux
        read-only
        root=/dev/xscsi/pci01.03.0-1/target1/lun0/part3

In the previous example, pay special attention to the append= line. That is, notice where the ioconfig information goes when using elilo.conf.

Persistent naming of Ethernet devices is an SGI proprietary mechanism and is supported on SGI Altix 3000 systems. Persistent naming refers to the mechanism that ensures that the Gigabit Ethernet card on the IO9 interface of an SGI Altix 3000 system is set up to always be eth0. It guarantees that the base Ethernet device number is assigned to the correct MAC address on an SGI Altix 3000 system even when multiple Ethernet devices are present in the system.

The /etc/sysconfig/networking/eth0_persist file contains the mapping of Ethernet device numbers to MAC addresses. If the file does not exist, it is created by the /etc/rc.d/init.d/eth_persist script, which is run at boot time. To ensure that eth0 is indeed assigned to the MAC address of the IO9 Ethernet card, it might be necessary to edit the file after the first time an SGI Altix 3000 system has been brought up after a clean install.

Besides ensuring that the mapping of Ethernet device numbers to MAC addresses persists as cards are added to a system, persistent naming also allows system administrators to control the way in which Ethernet devices are numbered. For example, if the Ethernet card with device number ethX is lost and the system administrator tries to recover by using the Ethernet card with device number ethY, it is possible to force the latter card to take on Ethernet device number ethX by editing the /etc/sysconfig/networking/eth0_persist file accordingly.

Following is a sample /etc/sysconfig/networking/eth0_persist file:

eth0 08:00:69:13:dc:ec
eth1 08:00:69:13:72:e8

The content of this file results in the following configuration:

[root]# ifconfig -a
eth0 Link encap:Ethernet  HWaddr 08:00:69:13:DC:EC
     inet addr:128.162.246.125 Bcast:128.162.246.255 Mask:255.255.255.0
     UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     RX packets:843 errors:0 dropped:0 overruns:0 frame:0
     TX packets:1245 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:100
     RX bytes:386044 (376.9 Kb)  TX bytes:126741 (123.7 Kb)
     Interrupt:59

eth1 Link encap:Ethernet  HWaddr 08:00:69:13:72:E8  
     BROADCAST MULTICAST  MTU:1500  Metric:1
     RX packets:136 errors:0 dropped:0 overruns:0 frame:0
     TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
     collisions:0 txqueuelen:100 
     RX bytes:8850 (8.6 Kb)  TX bytes:0 (0.0 b)
     Interrupt:63

If the slot directly to the right of the IO9 is populated by an Ethernet board, that is, bus 1, slot 2, and if you re-install Linux after it has been placed there or if you remove the eth0_persist persistent naming file, the IO9 could become eth1 instead of eth0. This is not recommended because it could affect your product licenses. If the Ethernet board is in any other IX-brick slot, you will not encounter this problem.

If it is necessary to have an Ethernet board in IO9 bus 1 slot 2, install the OS, including SGI ProPack, with the Ethernet board removed. After SGI ProPack is installed, you can replace the Ethernet board in bus 1 slot 2.

The SGI XSCSI subsystem on Linux leverages from IRIX functionality to provide more robust error handling, failover, and storage area network (SAN) infrastructure support as well as long-term large system performance tuning. XSCSI takes advantage of specific features of SGI architecture that standard open source drivers cannot without rewriting.

The naming convention of XSCSI device names is shown in the following example:

/dev/xscsi/pci01.03.0-1/target1/lun0/part1

Components of the XSCSI device name are as follows:

Notice that the device number (slot number), logical unit, and port number are fixed. These will never change. However, the system bus number could change because of a hardware problem (such as the I/O brick not booting) or a reconfiguration.

Persistent PCI-X bus numbering (see “ Persistent PCI-X Bus Numbering”), if enabled, provides persistent naming to prevent bus number changes even when the hardware fails or is reconfigured. If you use XSCSI names for mounting or locating devices and you also use persistent bus numbering, your XSCSI device names will always be persistent across reboots.

The XSCSI-SCSI subsystem provides SCSI device emulation of XSCSI devices. It allows programs written for the SCSI disk driver, the SCSI tape driver, and the SCSI generic driver to use devices controlled by the XSCSI drivers.

This section describes the IOC3 and IOC4 serial drivers, as follows:

/dev/ttyIOC3/0
/dev/ttyIOC3/1
/dev/ttyIOC3/2
/dev/ttyIOC3/3

/dev/ttyIOC3/line

crw-------    1 root     root      62,   1 Feb  7 18:11 0
crw-------    1 root     root      62,   2 Dec 31  1969 1
crw-------    1 root     root      62,   3 Feb  8 16:41 2
crw-------    1 root     root      62,   4 Dec 31  1969 3

crw-------    1 root     root      62,   1 Feb  7 18:11 0
crw-------    1 root     root      62,   2 Dec 31  1969 1
crw-------    1 root     root      62,   3 Feb  8 16:41 2
crw-------    1 root     root      62,   4 Dec 31  1969 3
crw-------    1 root     root      62,   5 Feb  7 18:11 4
crw-------    1 root     root      62,   6 Dec 31  1969 5
crw-------    1 root     root      62,   7 Feb  8 16:41 6
crw-------    1 root     root      62,   8 Dec 31  1969 7

/dev/ttyIOC4/0
/dev/ttyIOC4/1
/dev/ttyIOC4/2
/dev/ttyIOC4/3

For information on using standard Linux SCSI device drivers on XSCSI devices, see the Linux Configuration and Operations Guide.

The SGI XFS filesystem provides a high-performance filesystem for Linux. XFS is an open-source, fast recovery, journaling filesystem that provides direct I/O support, space preallocation, access control lists, quotas, and other commercial file system features. Although other filesystems are available on Linux, performance tuning and improvements leveraged from IRIX make XFS particularly well suited for large data and I/O workloads commonly found in HPC environments.

For more information on the XFS filesystem, see XFS for Linux Administration.

The SGI XVM Volume Manager provides a logical organization to disk storage that enables an administrator to combine underlying physical disk storage into a single logical unit, known as a logical volume. Logical volumes behave like standard disk partitions and can be used as arguments anywhere a partition can be specified.

A logical volume allows a filesystem or raw device to be larger than the size of a physical disk. Using logical volumes can also increase disk I/O performance because a volume can be striped across more than one disk. Logical volumes can also be used to mirror data on different disks.

This release adds a new XVM multi-host failover feature. For more information on this new feature and XVM Volume Manager in general, see XVM Volume Manager Administrator's Guide .

The SGI tape stream (TS) tape driver is low-level software that supports all of tape backup applications on SGI Altix systems. These applications include the Data Migration Facility (DMF), the Tape Management Facility (TMF), OpenVault, and xfsdump(8)/xfsrestore(8), the XFS filesystem incremental dump/restore utilities. For more information on the TS driver, see the Linux Configuration and Operations Guide.

The SGI Message Passing Toolkit (MPT) provides industry-standard message passing libraries optimized for SGI computers. On Linux, MPT contains MPI and SHMEM APIs, which transparently utilize and exploit the low-level capabilities within SGI hardware, such as its block transfer engine (BTE) for fast memory-to-memory transfers and the hardware memory controller's fetch operation (fetchop) support. Fetchops enable direct communication and sychronization among multiple MPI processes while eliminating the overhead associated with system calls to the operating system.

Parallel workloads, such as MPI jobs, can be launched, monitored, and controlled across a cluster or partitioned system using the SGI Array Services software. Array Services provides the notion of an array session, which is a set of processes that can be running on different cluster nodes or system partitions. Array Services is implemented using Process Aggregates (PAGGs), which is a kernel module that provides process containers. PAGGs has been open-sourced by SGI for Linux.

For more information on the Message Passing Toolkit, see the Message Passing Toolkit (MPT) User's Guide.

The SGI Performance Co-Pilot software was ported from IRIX to Linux to provide a collection of performance monitoring and performance management services targeted at large, complex systems. Integrated with the low-level performance hardware counters and with MPT, Performance Co-Pilot provides such services as CPU, I/O, and networking statistics; visualization tools; and monitoring tools.

For more information on Performance Co-Pilot, see the Performance Co-Pilot for IA-64 Linux User's and Administrator's Guide.

Programmable read-only memory ( PROM) chips are placed in your computer at the factory with software programmed into them that allows the CPU to boot and allows you to perform system administration and software installations. The PROM chips are not part of your disk or your operating system; they are the lowest level of access available for your system. You cannot erase them or bypass them. For more information on PROM and the L1 controller, see the SGI L1 and L2 Controller Software User's Guide.

SGI Altix 3000 systems provide the Extensible Firmware Interface (EFI), a supporting platform to provide input to the CPU and to handle its output. In addition, the EFI controls the server's boot configuration, maintaining the boot menu in durable, nonvolatile memory.

SGI ProPack 3 uses the elilo-3.3 package, which is fully compliant with EFI specification 1.10 with regard to where the bootloader (elilo.efi) should be located in the EFI system partition. According to that specification, the bootloader must be located in a dedicated vendor directory. On SGI Altix systems, that directory is /boot/efi/efi/sgi/. As of version 3.3, elilo looks for its configuration file only in the directory from which it is loaded. Because this also applies to the kernel, the kernel image, which was previously installed in /boot/efi/, is now installed in /boot/efi/efi/sgi/. For further documentation and details about this change, see the elilo documentation in /usr/share/docs/.

From the EFI prompt, you can perform basic file-management tasks (including text editing), make configuration changes, or write scripts that execute at boot time. For a summary of EFI commands, see Table 3-2.

FLEXlm is a flexible license management system from Macrovision that lets independent software vendors (ISVs) license their products and helps system administrators install and manage licenses with minimal overhead. It supports a wide range of licensing options, including simple node-locked licenses and floating licenses with redundant servers.

To build licensed software, ISVs must purchase a set of keys from Macrovision. System administrators can install license servers anywhere. Products purchased from SGI are typically licensed using FLEXlm.

For more information, visit

http://www.macrovision.com/solutions/esd/flexlm/flexlm.shtml

SGIconsole is a suite of software that allows you to manage multiple SGI servers running the IRIX or Linux operating system. These servers include SGI servers, SGI partitioned systems and large, single-system-image servers, including legacy SGI servers and Silicon Graphics graphical systems. SGIconsole can also provide console services for non SGI servers.

SGIconsole consists of software suite including the Console Manager package and Performance Co-Pilot, which provides access to common remote management tools for hardware and software. SGIconsole can manage SGI Altix and SGI Origin servers better than any other console server product. SGIconsole can communicate with L1, L2, module System Controller (MSC), and rackmount system multimodule System Controller (MMSC) system controllers and provide reset/nonmaskable interrupt (NMI)/power up/power down functionality for SGI servers.

Console Manager is a graphical user interface for the SGIconsole management and monitoring tool used to control multiple SGI servers. SGIconsole also has a command line interface. For more information on SGIconsole, see the SGIconsole Start Here.

This section describes the system controller firmware used in SGI Altix systems.

The L1 and L2 controllers are system controller firmware used in SGI systems.

The L1 controller is embedded in each brick in SGI Origin and Onyx 3000 series systems and in SGI Altix 3000 systems. It provides power and control sequencing, along with temperature and power monitoring for each brick.

The L2 controller is a rack-level controller that monitors and controls the bricks in its rack. All L2 controllers in a system are networked together and they consolidate the control and monitoring information from each brick to provide system-level control and monitoring.

The following references describe how to install and connect to an L1 or L2 controller:

For more information on the L1 and L2 system controller firmware, see the SGI L1 and L2 Controller Software User's Guide.

The L3 controller is a Linux software package that resides on a PC workstation or laptop computer. It provides remote support, system controller consoles, and system-level maintenance functions. The L3 controller software is obtained through the "System Controller Software 1.x" release.

The dlook(1) command displays the memory map and CPU use for a specified process. The following information is printed for each page in the virtual address space of the process:

Optionally, the amount of elapsed CPU time that the process has executed on each physical CPU in the system is also printed.

The dplace(1)  command binds a related set of processes to specific CPUs or nodes to prevent process migrations. In some cases, this tool improves performance because of the occurrence of a higher percentage of memory accesses to the local node.

For more information on NUMA tools, see chapter 4, “NUMA Tools” in the Linux Configuration and Operations Guide and chapter 5, “Data Placement Tools” in the Linux Application Tuning Guide.

OpenGL Multipipe enables owners of high performance visualization solutions from Silicon Graphics to create high-resolution SGI Reality Center environments and achieve higher levels of performance while continuing to use existing single pipe applications. It brings the power of scalable visualization to new applications and enables organizations to create better products, generate new ideas, and increase operational efficiency.

OpenGL Multipipe enables high levels of screen space scaling and low-to-moderate levels of performance scaling without application modification. OpenGL Multipipe uses a flexible architecture that enables a logical screen to be built up from from individual physical screens that can be located anywhere within the logical screen space. This enables the creation of large walls with 2D arrays of displays, wide panoramic screens with overlapping displays for edge blending and the use of a scalable graphics compositor to drive a single screen with increased performance. For more information about OpenGL Multipipe, visit http://www.sgi.com/products/software/multipipe/ .

OpenGL Multipipe SDK is an API layer for OpenGL that provides a simplified approach to managing graphics applications across multiple graphics subsystems or pipes and focused on delivering run-time scalability on SGI scalable graphics servers.

Applications written using the OpenGL Multipipe SDK API can run seamlessly from desktop single-processor, single-pipe systems up to large multiprocessor, multipipe scalable graphics systems. A simple configuration file defines the desired configuration of the application, such that the same application can support normal office environments, SGI Reality Center rooms, walls, immersive desk displays, and domes. In addition, the same application can take advantage of scalable graphics systems by combining the power of multiple graphics pipes into a single display output, which in turn can be scaled to drive multiple display devices. OpenGL Multipipe SDK provides a straightforward solution for graphics applications that were designed for single-processor, single-pipe applications to scale seamlessly when run on graphics systems with multiple processors and multiple graphics pipes. For more information about OpenGL Multipipe SDK, visit http://www.sgi.com/products/software/multipipe/sdk/ .

OpenGL Performer is a powerful and comprehensive programming interface for developers creating real-time visual simulation and other professional performance-oriented 3D graphics applications. The toolkit simplifies development of applications used for visual simulation, manufacturing, simulation-based design, virtual reality, scientific visualization, interactive entertainment, broadcast video, architectural walk-through, and computer aided design.

The latest release is built atop the industry standard OpenGL graphics library, interoperates with OpenGL Volumizer, OpenGL Multipipe SDK, and OpenGL Vizserver, includes both ANSI C and C++ bindings, and is available for the IRIX operating system, and the Linux, Windows XP, and Windows 2000 operating systems. For more information about OpenGL Performer, visit http://www.sgi.com/software/performer/.

OpenGL Vizserver is a technical and creative computing solution designed to deliver superior visualization, collaboration functionality, and advanced performance to any client, whether on a desktop workstation or wireless tablet. OpenGL Vizserver allows users to remotely view and interact with large data sets from any other system at any location in an organization and to collaborate with multiple colleagues using these same applications and data.

OpenGL Vizserver enables a single SGI advanced visualization system to act as a multiuser graphics server which distributes remote and collaborative visualization sessions to client systems running a mix of desktop operating systems. With OpenGL Vizserver, graphics processing is handled entirely on the SGI advanced visualization system using full 3D hardware acceleration; the client machine simply decompresses and displays the image. This leverages existing internal infrastructure and allows customers to leverage the power of the world-class SGI advanced visualization systems on common desktop workstations such as Silicon Graphics visual workstations, Sun Solaris systems or Intel architecture PCs running the Linux, Microsoft Windows NT, Windows 2000, or Windows XP operating systems. For more information about OpenGL Vizserver, visit http://www.sgi.com/products/software/vizserver/ .

OpenGL Volumizer is a cross platform, high-level volume rendering API for the energy, manufacturing, medical, and sciences markets. OpenGL Volumizer is a revolutionary graphics API designed for interactive, high quality, scalable visualization of large volumetric data sets. OpenGL Volumizer provides a high-level interface to OpenGL hardware to allow application writers and researchers to visualize multiple gigabytes of volumetric data. The API uses OpenGL for volume rendering and hence allows standard graphics applications to treat volumetric and surface data in a similar fashion.

OpenGL Volumizer is a library of C++ classes that facilitates the manipulation and display of volumetric data sets common in geo-science, medical, scientific and engineering applications. It provides a layer of functionality that sits on top of OpenGL and integrates seamlessly into higher-level toolkits and applications. For more information on OpenGL Volumizer, visit http://www.sgi.com/products/software/volumizer/ .