Chapter 13. Adding an Execution Node to the Cluster

If you later want to add an execution node to the cluster, you can manually install packages on the node by using the rpm(8) command or you can use the installace utility as follows to install software on just that node:

For more information about installace, see “Interactive Mode Instructions” in Chapter 4, and “Batch Mode Instructions ” in Chapter 4.

Configuration Tasks

After the software is installed, perform the following configuration tasks on the new execution node and head node, according to the products you are using.

New Execution Node

Do the following on the new execution node:

  • Ethernet MPICH:

    • Add all cluster node names to the /usr/share/machines.LINUX file

  • Myrinet GM-MPICH:

    • Add all cluster node names to the /usr/util/machines/machines.LINUX file

  • PBS:

    • Edit the /usr/spool/pbs/mom_priv/config file and add the following:

      $logevent 0x1ff
      $clienthost headnode

    • Edit the /usr/spool/pbs/server_name file so that it contains the name of the head node.

    • Start the pbs_mom process by entering the following:

      # /usr/local/sbin/pbs_mom

  • PCP:

    • Start the pmcd daemon:

      # /etc/rc.d/init.d/pcp start

  • Lconsole:

Head Node

Do the following on the head node:

  • All products:

    • Edit the /etc/ace/nodes file to include the name of the new execution node.

  • Myrinet GM-MPICH:

    • Edit the gm configuration files ($HOME/.gmpi/confand /etc/ace/gmpi.conf by default) according to the directions in Chapter 7, “Configure gm-mpich”.

    • Add the node name to the /usr/util/machines/machines.LINUX file.

    • Tell users to copy the an existing gm configuration file (such as $HOME/.gmpi/conf).

  • Ethernet MPICH:

    • Add the node name to the /usr/share/machines.LINUX file.

  • PBS:

    • Edit the /usr/spool/pbs/server_priv/nodes file and add the name of the new execution node.

    • Start the pbs_mom process by entering the following:

      # /usr/local/sbin/pbs_mom

    • If you wish to perform a PBS check, enter the following :

      # /usr/local/bin/pbsnodes -a

  • Lconsole:

Interactive Mode Example

The following example shows user entries in bold:

root@ace07 /root]# /bin/mount /dev/cdrom /mnt/cdrom
[root@ace07 /root]# cd /mnt/cdrom
[root@ace07 cdrom]# ./installace

ACE nodes file (/etc/ace/nodes) already exists.
If you continue with the install, you will be redefining your cluster.
CONTROL-C if you want to stop this install.


    Advanced Cluster Environment v. 1.4 for Linux i686 includes the
    following products:

        pbs       PBS batch-queuing system
        pcp       Performance Co-Pilot
        mpich     MPICH message-passing library - ethernet
        gm-mpich  Myricom's MPICH message-passing library
        gm        Myricom's GM driver
        lconsole  Command-line serial console
        hoover    Hoover cluster manager & serial console

    Enter "ethernet" to select:
        mpich pbs pcp lconsole

    Enter "myrinet" to select:
        gm gm-mpich pbs pcp lconsole

    Or enter the individual products separated by white space.

Please enter all desired products separated by white space.
ACE> eth

    Note:  When providing a hostname, use the fully qualified name
    such as "foo.domain.com". If the hostname is resolved on all
    nodes, you can abbreviate it to "foo".

    The installace utility can modify hosts.equiv files on your nodes.
    These files are used by MPICH, GM-MPICH, and PBS.

    You can choose to provide the information required to edit these files
    or avoid this action by entering the appropriate number:

         1)  Enter the hostnames individually
         2)  Specify a file containing the list of hostnames
         3)  Specify a pattern for hostnames (NAME1.domain.com)
         4)  Install software only on this node

Enter choice (1, 2, 3, or 4).
ACE> 1
    Enter the hostname of the head node.
    The head node is the host from which users will launch
    execution of parallel jobs.

Enter the hostname of the head node.
ACE> .

    Enter the hostnames for each execution node in the
    cluster.  After you have entered all hostnames,
    Enter '.' or <control-D> on a new line to end the list.

ACE> ace06
ACE> .

    The install process can update the /etc/hosts.equiv file on all
    the hosts in the cluster.  The /etc/hosts.equiv will allow access
    to the hosts.  If /etc/hosts.equiv file is not being used, each user
    must update their .rhosts file to allow rcp/rsh type access to all
    cluster nodes.  Enter one of the following:

      1) Add the specified hostnames to the existing /etc/hosts.equiv files.
         If the file does not exist, create it.
      2) Replace the current /etc/hosts.equiv file with a new one
         containing only the specified cluster hostnames.  The current
         /etc/hosts.equiv will be saved as /etc/hosts.equiv.old.
      3) Do not change the existing /etc/hosts.equiv file.

Enter choice (1, 2, or 3).
ACE> 1

    Checking rsync/rcp/rsh access ...

    Checking disk space requirements ...

    Installing selected ACE products on node(s)
      ace-tools ........... [yes]
      ace-docs ............ [yes]
      pcp ................. [yes]
      pbs ................. [yes]
      mpich ............... [yes]
      gm .................. [no]
      gm-mpich ............ [no]
      lconsole ............ [yes]
      hoover............... [no]


    Installing execution node software on ace06
    Installing package mpich-1.2.1-1 on ace06 ...
MPICH postinstall script
    Installing package pcp-2.1.9-12 on ace06 ...
    Installing package pcp-pro-2.1.5-2 on ace06 ...
    Installing package pcp-ace-1.3.0-4 on ace06 ...
Creating MPI wrapper
...Done, output in /var/tmp/pcp-ace-1.3.0-4-mpi_wrapper
    Package textutils-2.0e-3 already installed on ace06, skipping.
    Installing package pbs-mom-2.2-11RH6 on ace06 ...
PBS mom preinstall script
PBS mom postinstall script

    Configuring selected ACE products on node(s) ...
    Configuring PBS on ace06 ...
PBS has not been configured on the head node for
this new execution node.
You must do the following to properly configure PBS:
1) Add the name of the head node to ace06:/usr/spool/pbs/server_name
2) Edit ace06:/usr/spool/pbs/mom_priv/config to add the following:
$logevent 0x1ff
$clienthost 3) Edit the head node /usr/spool/pbs/server_priv/nodes file and
add the execution node ace06.

    Starting PCP daemons ...
    Starting PCP on ace06 ...

    Post configuration ...
    Adding to the /etc/hosts.equiv file on all nodes ...

    This ACE release also contains third-party evaluation
    software. See /mnt/cdrom/ace-demos/README.TXT for
    more information.

    The installation/configuration process has been completed!

Batch Mode Example

This section shows an example of a modified install.conf file and output.

Modified install.conf File

The following example shows user changes (from the template /mnt/cdrom/install.conf file) in bold:

#
# This file contains Advanced Cluster Environment (ACE)
# configuration information that can be used by installace.
#
# This file must be /bin/sh parsable.
#
#
# Advanced Cluster Environment for Linux, includes the
# following products:
#
#       pbs       PBS batch-queuing system
#       pcp       Performance Co-Pilot
#       gm        Myricom's GM driver
#       gm-mpich  Myricom's enhanced MPICH message-passing library
#       mpich     MPICH message-passing library
#       lconsole  Command-line serial console
#       hoover    Hoover cluster manager & serial console
#
#    Enter "ethernet" to select:
#        mpich pbs pcp lcdinc
#
#    Enter "myrinet" to select:
#        gm gm-mpich pbs pcp lcdinc
#
# List all desired products (separated by commas or white space)
# in the 'Products' variable.

Products="eth"


#
# ACE needs to know about all the nodes in the cluster.
# There are two ways to specify this information:
# 1) Create a separate file containing all the nodes in the cluster.
#    The first node listed must be the head node.
#    If you want to use this option, list the full pathname in
#    he 'Nodes_filename' variable below.
#
# 2) Define the head and execution nodes in this configuration file.
#    To do this, fill in the 'Head_node' and the 'Execution_nodes'
#    variables below.
#

#
# List the full path to the file containing all cluster nodes.
# The first node listed must be the head node.
# Leave empty if you want to specify nodes below.
#

Nodes_filename=""


# List the head node.  The head node is the host from which
# users will submit PBS requests and launch the execution of parallel
# jobs.  Use the fully qualified name such as "foo.domain.com".
#
# If you have left 'Nodes_filename' variable empty and you do not
# fill in a node for 'Head_node', only the software appropriate to
# execution nodes will be installed.  In this case, you must manually
# configure PBS on the head node for the execution nodes.

Head_node=""


#
# NOTE: If you have left 'Nodes_filename' variable empty and
#       you do not list a node for 'Head_node', only the software
#       appropriate to execution nodes will be installed.  In this case,
#       you must manually configure PBS for each new execution node
#       by doing the following:
#
#       1) Add the name of the head node to /usr/spool/pbs/server_name
#          on the execution nodes.
#       2) Edit /usr/spool/pbs/mom_priv/config on the execution nodes
#          to add the following:
#              $logevent 0x1ff
#              $clienthost
#       3) Edit the /usr/spool/pbs/server_priv/nodes file on the head node
#          and add the name of the execution nodes.

#
# List all execution nodes in the cluster.  They can be comma or
# white space, including newline, separated.
# Use the fully qualified name such as "foo.domain.com".
#
# If you want the head to also be an execution node, you
# must also list it in the 'Execution_nodes' variable.
#

Execution_nodes="ace06"


#
# The install process can update the /etc/hosts.equiv file on all
# the nodes in the cluster.  The /etc/hosts.equiv file allows access
# to the nodes.  The MPICH, PBS, and GM products use rsh/rcp to access
# other nodes.  If /etc/hosts.equiv is not updated, all users will
# have to update their .rhosts file on all nodes.
# Set the 'Hosts_equiv_file' variable below to one of
# the following values:
#
#      "add") Add the specified nodes to the existing /etc/hosts.equiv files.
#             If /etc/hosts.equiv does not exist, installace will create one.
#  "replace") Replace the current /etc/hosts.equiv file with a new one
#             containing only the specified nodes.  The current
#             /etc/hosts.equiv will be saved as /etc/hosts.equiv.old.
# "nochange") Do not change the existing /etc/hosts.equiv file.
#             If the variable is not set, the default will be "nochange".

Hosts_equiv_file="add"

Batch Output

The following shows the output when using the above file:

[root@ace08 cdrom]# ./installace /tmp/install.conf


ACE nodes file (/etc/ace/nodes) already exists.
If you continue with the install, you will be redefining your cluster.
CONTROL-C if you want to stop this install.

Using install config file '/tmp/install.conf'.

    Checking rsync/rcp/rsh access ...

    Checking disk space requirements ...

    Installing selected ACE products on node(s)
      ace-tools ........... [yes]
      ace-docs ............ [yes]
      pcp ................. [yes]
      pbs ................. [yes]
      mpich ............... [yes]
      gm .................. [no]
      gm-mpich ............ [no]
      lconsole ............ [yes]
      hoover .............. [no]



    Installing execution node software on ace06
    Installing package mpich-1.2.1-1 on ace06 ...
MPICH postinstall script
    Installing package pcp-2.1.9-12 on ace06 ...
    Installing package pcp-pro-2.1.5-2 on ace06 ...
    Installing package pcp-ace-1.3.0-4 on ace06 ...
Creating MPI wrapper
...Done, output in /var/tmp/pcp-ace-1.3.0-4-mpi_wrapper
    Package textutils-2.0e-3 already installed on ace06, skipping.
    Installing package OpenPBS-mom-2-3-mom on ace06 ...
PBS mom preinstall script
PBS mom postinstall script

    Configuring selected ACE products on node(s) ...
    Configuring PBS on ace06 ...
PBS has not been configured on the head node for
this new execution node.
You must do the following to properly configure PBS:
1) Add the name of the head node to ace06:/usr/spool/pbs/server_name
2) Edit ace06:/usr/spool/pbs/mom_priv/config to add the following:
$logevent 0x1ff
$clienthost 3) Edit the head node /usr/spool/pbs/server_priv/nodes file and
add the execution node ace06.

    Starting PCP daemons ...
    Starting PCP on ace06 ...

    Post configuration ...
    Adding to the /etc/hosts.equiv file on all nodes ...

    This ACE release also contains third-party evaluation
    software. See /mnt/cdrom/ace-demos/README.TXT for
    more information.

    The installation/configuration process has been completed!