This chapter highlights possible solutions to common problems.
If the installace script does not have rsh(1) access, it may be because a /etc/securetty file exists. Move the file to /etc/securetty.save and rerun installace.
If you see errors related to .rhosts in /var/log/messages, it may be because the file permissions are not set properly. Change the file permissions so that group and other do not have write permission.
For example:
# chmod 744 .rhosts |
If you run into problems with pluggable authentication modules (PAM), you may need to do one or more of the following:
For more information about PAM, see the Red Hat Linux 6.0 The Official Red Hat Linux Installation Guide.
This section describes solutions to common MPICH problems.
If you see a Permission denied message from mpirun(1), it probably means that the user does not have permission to use rsh(1) to start the process. To test for access, use the confidence(1) script. For example:
# /usr/local/ace/bin/confidence |
If you have installed mpich, you can also use the tstmachines command:
$ /usr/sbin/tstmachines |
The script will display an error and corrective actions if it fails. For more information, see the tstmachines(1) man page and the User's Guide for mpich, a Portable Implementation of MPI.
If you see a No such file or directory error when running mpirun(1), make sure that the mpi binary is accessible on all nodes in the cluster, at the same location. This can be done by using an NFS mount an an identical path, or by copying the binary to each node in the cluster, using an identical path on each node.
The /usr/share/machines.LINUX file contains the nodes configured for use by MPICH. The entries should be listed one node per line. If the node name has the :# suffix, it shows the number of MPI processes supported on that node at one time. The number is usually the same as the number of CPUs. The mpirun command distributes the MPI process in a orderly manner.
For example:
node1:2 node2:2 node3:2 |
The mpirun command will put two MPI processes on node1 before putting any processes on node2. If you want to spread the load out across the nodes, you could edit the file to look like the following:
node1:1 node2:1 node3:1 node1:1 node2:1 node3:1 |
Both formats support 6 processes.
For a temporary fix, users can create their own a file using this format and run it in place of the default /usr/share/machines.LINUX by specifying it using the following option:
mpirun -machinefile mymachinefile |
This section lists a few common problems when using PBS. For more information, see the PBS Administrator's Guide.
To make an head node an execution node, do the following:
Edit the /usr/spool/pbs/server_priv/nodes file and remove the :ts suffix from the node name.
For example, where ace04 is the node name:
Old: ace04:ts np=2
New: ace04 np=2
Restart PBS by entering the following:
# /etc/rc.d/init.d/pbs restart |
Verify your change by entering the following:
# /usr/local/bin/pbsnodes -a |
If the output for head node contains the following, the head node is now an execution node:
ntype = cluster |
To make a node that is both a head node and an execution node into just a head node, do the following:
Edit the /usr/spool/pbs/server_priv/nodes file and add the :ts suffix from the node name.
For example, where ace04 is the node name:
Old: ace04 np=2
New: ace04:ts np=2
Restart PBS by entering the following:
# /etc/rc.d/init.d/pbs restart |
Verify your change by entering the following:
# /usr/local/bin/pbsnodes -a |
If the output for head node contains the following, the head node is strictly a head node and is no longer also an execution node:
ntype = time-shared |
If PBS jobs fail to start, it is likely that the execution nodes cannot communicate with the head node. Do the following:
If the root user is prevented from submitting jobs, use another user ID. The default installed ACE PBS will not be configured so that root can submit requests.
To configure PBS to allow root to submit requests, execute the following on the head node:
# /usr/local/bin/qmgr -c 'set server acl_roots="root@*" ' |
To get PBS state information on all nodes, execute the following on the head node:
# /usr/local/bin/pbsnodes -a |
If a node is missing, add it to the /usr/spool/pbs/server_priv/nodes file and restart PBS by entering the following on the head node:
# /etc/rc.d/init.d/pbs restart |
If a node is down, ensure that the execution node's /usr/spool/pbs/server_name file contains the name of the head node. You must restart pbs_mom by entering the following on the execution node:
# /etc/rc.d/init.d/pbs_mom restart |
It may take a few moments for the head node to synchronize up with the execution node.
PCP applications will return the following error if the PCP daemons have died or have not previously been started:
application: hinv.ncpu: Unknown metric name application: Failed to get the number of CPUs from host "test" |
To resolve this problem, restart the PCP daemons as follows:
# /etc/rc.d/init.d/pcp stop # /etc/rc.d/init.d/pcp start |
The PCP daemons must be running on every execution node in the cluster (optionally on the head node).
For information about other problems, see the following files:
/usr/doc/pcp-Version/README
/usr/doc/pcp-ace-Version/README
/usr/doc/pcp-pro-Version/README
The nfs(5) manpage incorrectly says that the NFS version defaults to version 2; it actually defaults to version 3. This is a problem when integrating NFS into a standard Linux environment. To get around this problem, you must mount the other Linux filesystems as NFS version 2 by using nfsvers=2 in the /etc/fstab file.
For example:
voila:/users /users nfs exec,dev,suid,rw,nfsvers=2 1 1 |
For more information, see the fstab(5) man page.