Chapter 9. Upgrading an Existing Cluster
Prev   Next

Chapter 9. Upgrading an Existing Cluster

To upgrade an existing cluster, do the following:

  1. Wait until all PBS running requests have completed.

  2. Save the PBS server and queue structures. For example, enter the following:

    # qmgr -c "print server" > /tmp/server.dat

  3. Shut down the cluster services:

    1. Stop the following PBS daemons on the head node:

      # crsh "uname -n; killall -9 pbs_server"
# crsh "uname -n; killall -9 pbs_sched"

    2. Stop the the following PBS daemon on the execution node and on the head node if so configured:

      # crsh "uname -n; killall -9 pbs_mom"

    3. Stop the PCP daemon on all nodes:

      # crsh "uname -n; killall -TERM pmcd"

    4. For Myrinet clusters:

      1. Stop gmapper:

        # killall -9 gmmapper

      2. Stop any Myrinet interfaces. For example:

        # ifconfig myri0 down

      3. If there are outstanding references to the driver, unload it. For example:

        # lsmod gm
Module                  Size  Used by
gm                    257784   0  (unused)

# rmmod gm

  4. Save the following configuration files and directories:

    • Head node:

      • /etc/ace/nodes

      • /etc/ace/manifest

      • /etc/ace/gmpi.conf

      • /usr/spool/pbs/server_name

      • /usr/spool/pbs/server_priv/nodes

      • /usr/spool/pbs/mom_priv/config

      • /var/Lconsole/.IC

      • /usr/lib/vacm/vacm_configuration

    • Each execution node:

      • /usr/spool/pbs/server_name

      • /usr/spool/pbs/server_priv/nodes

      • /usr/util/machines/machines.LINUX (for gm-mpich and for mpich ACE 1.0)

      • /usr/share/machines.LINUX (for mpich ACE 1.1 and later)

  5. If you are using Myrinet, do one of the following:

  6. Upgrade the ACE software by using the installace utility. See Chapter 4, “Installing ACE Software”. You can use the /etc/ace/nodes file that you copied in the previous step (such as /etc/ace/nodes.old) as input to the host file question.

  7. Reconfigure PBS. For example:

    # qmgr < /tmp/server.dat

  8. Restore the saved configuration files and directories

  9. For Myrinet:

    1. Reload Myrinet drivers: 

      # cd /usr/local/gm
# scripts/gm_install

    2. Start the mapper process:

      # scripts/gmmapper start

      Note: The Myrinet RPM writes the driver startup to the /etc/rc.d/rc.local script.

  10. For Ethernet:

    1. Move the saved copy of /usr/util/machines/machines.LINUX to /usr/share/machines.LINUX .

  11. Run the confidence tool. See Chapter 8, “Verify the Installation”.

Prev Table of Contents Next
Chapter 8. Verify the Installation  Chapter 10. Using and Enhancing Performance Co-Pilot (PCP)