This document discusses the Linux Advanced Cluster Environment (ACE) software. It explains how to perform configuration, installation, and administrative tasks.
 | Note: Examples in this document may not show the current RPM numbers. |
This document uses the following terminology:
Node, which is an SGI 1400 or SGI 1200 computer. A node is a host.
Head node, which is the node that performs administrative tasks for the cluster and is where PBS requests are submitted. Users log into the head node. A head node can also be an execution node, but there can be only one head node.
Cluster, which is made up of one head node and multiple execution nodes.
Remote administration node, which is the node from which Lconsole administrative tasks are performed.
Serial multiplexer, which is a hardware and software product that allows a remote machine to perform administrative tasks . The Digi International EtherLite 16 is the default product available from SGI; the Comtrol RocketPort is an alternative product required for use with the Hoover cluster manager.
Linux ACE supports throughput and capability clusters:
The instructions in this document assume the following:
You know the IP addresses and hostnames of each node in the cluster. (The hostname is the output of the hostname(1) command.)
The base Linux operating system (OS) -- Red Hat version 6.2, SuSE version 6.4, or TurboLinux 6.0/6.0.2 or later - has already been installed on the systems in your cluster. For more information, see the base OS documentation.
This section discusses:
Linux ACE contains the following products:
An administration tool (mutually exclusive):
Lconsole command-line serial console
Hoover cluster manager and serial console
Portable batch system (PBS)
Performance Co-Pilot (PCP)
Message Passing Interface (MPI) software packages that supports parallel programming across a network of computer systems through a technique known as message passing (mutually exclusive):
MPICH message-passing library for Ethernet
Myricom's MPICH message-passing library and Myricom's GM driver
Drivers for the EtherLite 16 (EL16) serial multiplexer
Extra software
Evaluation software
You will use the installace utility to install the packages on the head and execution nodes you specify.
This section discusses Lconsole and Hoover.
The Lconsole utility provides a serial port console that lets you do the following:
Use serial lines to get the boot prompt in case of a network failure
Perform a remote reset in case of hardware failure
| Note: Lconsole requires that you use a serial multiplexer. You should not install both Lconsole and Hoover. |
For information about using Lconsole, see Chapter 11, “Lconsole Utility”.
Hoover is a graphical user interface (GUI) for administration tasks on the nodes or cluster. It lets you monitor node status and perform certain system administration functions such as the following:
Power cycle or reset the node
View the System Event Log (SEL)
Open a serial console to the node
By default, Hoover is organized into two tabbed panels: the top panel is the status and navigation window and the bottom panel contains data panels that result from actions in the top window, such as requesting a SEL for a node.
You should not install both Lconsole and Hoover.
| Note: If you intend to use Hoover, you should install the Comtrol RocketPort serial port concentrator, not the Digi EtherLite 16 device. |
The Hoover binaries are located in /usr/bin.
For more information, see the following:
/usr/doc/vacm-hoover-version/
/usr/doc/vacm-version/
The Portable Batch System (PBS) is developed by Veridian Systems.
The binaries for PBS are installed in the /usr/local/sbin and /usr/local/bin directories. The configuration files are in the /usr/spool/pbs directory.
For more information, see the following:
PBS Administrator's Guide found in /usr/doc/OpenPBS-version/pbs_admin_guide.pdf
Performance Co-Pilot (PCP) is a framework and services to support system-level performance monitoring and performance management. The PCP infrastructure provides a unifying abstraction for all of the interesting performance data in a system. It also allows client applications to easily retrieve and process any subset of that data.
PCP utilizes a client-server architecture to provide the following:
Tools for visualizing all hosts in the cluster. For details, see the manual pages for clustervis(1) and pmgcluster(1).
Tools for instrumenting MPI applications so that they can export MPI library call counters and other performance statistics into the PCP framework, and a tool for visualizing this data. For details, see the manual pages for pcp_mpi(3), mpimon(1), mpivis(1), pmdampi(1), and pmfindash(1), and the examples in the /usr/share/pcp/demos/pcp_mpi directory.
A PCP agent for extracting performance statistics from one or more Myrinet switches and exporting this into the PCP framework, and a tool for visualizing this data. For details, see the manual pages for myrinetmon(1), pmdamyrinet(1) and the /var/pcp/pmdas/myrinet/README file.
Centralized real-time monitoring of the performance on distributed nodes
Extensible collection framework where new sources of performance data can be added using a plug-in architecture
Monitoring tools providing textual, 2-D and 3-D graphical interfaces
Intelligent rule-based filtering may be used to implement alarms and automated monitoring
Integrated archive logging and replay for all monitoring tools
The binaries for PCP are installed in /usr/bin (public commands) and /usr/share/pcp/bin (administrative and private commands). Configuration files reside in /var/pcp and log files are in /var/log/pcp. The file /etc/pcp.conf describes in greater detail where the components of the PCP installation are installed.
For more information, see http://oss.sgi.com/projects/pcp.
The version of PCP packaged with Linux ACE includes features that are derived from the IRIX version of PCP. For more information, see: http://www.sgi.com/software/co-pilot/
Detailed instructions for using and configuring the PCP monitoring tools are available in the Performance Co-Pilot User's and Administrator's Guide. Users who wish to develop their own performance monitoring tools and/or their own PCP agents, should read the Performance Co-Pilot Programmer's Guide. Both of these books are available online from the SGI Technical Publications Library: http://techpubs.sgi.com/
These documents describe the IRIX version of the PCP product, which differs only slightly from the Linux version.
Table 1-1. IRIX and Linux Differences
Directory | IRIX | Linux |
|---|---|---|
rc/startup scripts | /etc/init.d | /etc/rc.d/init.d |
Private PCP binaries | /usr/pcp/bin | /usr/share/pcp/bin |
Shared PCP files (shareable for diskless) | /usr/pcp | /usr/share/pcp |
Directory of manual pages | /usr/share/catman | /usr/man |
PCP logs | /var/adm/pcplog | /var/log/pcp |
PCP documentation | /var/pcp | /usr/doc/pcp-Version |
Directory for PCP demos and examples | /var/pcp/demos | /usr/share/pcp/demos |
magic, as used by file(1) | /etc/magic | /usr/share/magic |
Source code examples for using PCP are installed in the /usr/share/pcp/demos directory. In particular, the /usr/share/pcp/demos/pcp_mpi directory contains an example MPI application that has been instrumented to export MPI library call counts and statistics into the PCP framework. Once this is set up, you can use the mpivis(1) tool to visualize the MPI call rates and performance of MPI applications that have been instrumented in this way. For detailed instructions, see the mpivis(1), pmfindash(1), and pcp_mpi(3) manual pages.
For a summary of the differences between this version of PCP and the previous release, see the following files:
/usr/doc/pcp-version/CHANGELOG
/usr/doc/pcp-pro-version/CHANGELOG
/usr/doc/pcp-pro-version/CHANGELOG
You can only install one of the following:
MPICH message-passing library for Ethernet
Myricom's enhanced MPICH message-passing library and Myricom's GM driver
Binaries are located in the /usr/bin directory. Header files are located in the /usr/include directory.
The Ethernet and Myricom versions use the same binaries, header files, and documentation, but they have different libraries:
Ethernet MPICH libraries are in /usr/build/LINUX/ch_p4/lib
Myricom MPICH libraries are in /usr/build/LINUX/ch_gm/lib
| Note: Installing mpich or gm-mpich will overwrite some of the header files used by lam, another MPI product. |
Ethernet MPICH is produced by Argonne National Laboratory.
For more information, see the following:
/usr/doc/mpich-version/install.ps.gz for the Installation Guide for mpich, a Portable Implementation of MPI
/usr/doc/mpich-version/guide.ps.gz for the User's Guide for mpich, a Portable Implementation of MPI
/usr/doc/mpich-version/index.html for online help
The Myricom MPICH release has two products from Myricom:
MPICH
gm drivers
For more information, see the following:
/usr/doc/mpich-version/install.ps.gz for the Installation Guide for mpich, a Portable Implementation of MPI
/usr/doc/mpich/version/guide.ps.gz for the User's Guide for mpich, a Portable Implementation of MPI
/usr/doc/mpich/version/www/index.hmlt for online help
SGI SystemImager enables system administrators to easily replicate an entire Linux system onto a set of Linux machines, even when the machines have unformatted and unpartitioned disks. The GUI makes it easy to keep the hard drives of those systems synchronized after initial replication.
If you are a system administrator who manages mostly identical machines, you will find the SystemImager tool useful in any or all of the following situations:
You are managing a new cluster with preinstalled or nonpreinstalled machines.
You need to add preinstalled or non-preinstalled machines to a cluster.
You need to recover a machine or group of machines from a crashed hard disk.
You are installing or upgrading software.
You need to change configurations (for example, to modify system files).
You need to resynchronize clients after files have been added or deleted.
For more information, see SGI SystemImager Software Installationand Administration Guide. To view this document, install the documentation RPM files found in /mnt/cdrom/ace-extras/systemimager. They will be installed in /usr/share/doc/ace-extras/sgi-SystemImager-version/SIGUI_AG.
The SGI SystemImager tool is based on the VA SystemImager product. For more information on VA SystemImager, see the following site:
http://www.systemimager.org/ |
ACE provides additional software on an evaluation basis. These products are made available for evaluation purposes and as such require procurement of a evaluation license from the respective companies. Agreement has been made with each company to provide at least a 30-day evaluation license.
| Note: SGI does not provide support for evaluation software. |
For details, see the following file:
/mnt/cdrom/ace-demos/README.TXT |
To perform the installation, you must have sufficient amounts of free space in /tmp and /usr on each node in the cluster. For example, to install the ethernet or myrinet package sets, you should have at least the following amounts of free space on each node in the cluster:
Table 1-2 shows the amount of space required by each package.
| Note: You will install one of the following:
If you use Myrinet, you should also install the gm driver package. |
Table 1-2. Approximate Space Requirements Per Package (in Mbytes)
RPM Package | Head Node | Execution Node |
|---|---|---|
gm | 3.5 | 3.5 |
gm-mpich | 1.7 | 1.7 |
hoover | 2.3 | (none) |
lconsole | 0.2 | (none) |
mpich | 2.2 | 2.2 |
pbs-client (mom) | (none) | 7.7 |
pbs (server) | 13.0 | (none) |
pcp | 2.0 | 2.0 |
pcp-ace | 0.2 | 0.2 |
pcp-pro | 11.3 | 11.3 |
sgi-ace-tools | 0.02 | (none) |
sgi-acedocs | 1.1 | (none) |
hoover | 2.5 | (none) |
sysimager | 23 | (none) |
Ensure that you have enough disk space before installing Linux ACE.
The following sections provide more information about the products supplied with ACE. You should use the individual product documentation found in /usr/doc. However, you do not need to perform individual product installations because the products are installed as part of the Linux ACE installation procedure. You will install individual RPMs only if you want to customize the installation.