Chapter 5. Troubleshooting

This chapter highlights possible solutions to common problems.

No rsh/rcp Access

If you are denied access through rcp(1) or rsh(1), one of the following remedies might work for you.

Remove /etc/securetty File

If the install script does not have rsh or rcp access, it may be because a /etc/securetty file exists. Move the file to /etc/securetty.save and rerun install.

Change File Permissions

If you see errors related to .rhosts in /var/log/messages , it may be because the file permissions are not set properly. Change the file permissions so that group and other do not have write permission, as in the following example:

# chmod 600 .rhost

Check PAM Authentication

If you run into problems with pluggable authentication modules (PAM), you may need to do one or more of the following:

  • If you are using '+' as a wild host name in the .rhosts file and the hosts are not being recognized as expected, check the pam.d/rshd file entry in the pam_rhosts_auth file and ensure that you have specified promiscuous in that entry. This specification must be present for the hosts marked '+' to be granted access.

  • Change the rhost.auth.so entry from required to sufficient.

    If authentication fails, the required control token specifies that PAM will call all other modules before denying authentication.

    If authentication is successful, the sufficient control token specifies that PAM will grant authentication even though a previously required module failed.

  • Reorder the auth lines in the /etc/pam.d/rlogin and /etc/pam.d/rsh files as follows:

    • Original order:

      auth required /lib/security/pam_securetty.so
      auth sufficient /lib/security/pam_rhosts_auth.so promiscuous

    • Corrected order:

      auth sufficient /lib/security/pam_rhosts_auth.so promiscuous
      auth required /lib/security/pam_securetty.so

For more information about PAM, see the Red Hat Linux 6.0 The Official Red Hat Linux Installation Guide .

PBS Jobs Fail to Start

If PBS jobs fail to start, it is likely that the execution nodes cannot communicate with the head node. Do the following:

  • Check that pbs_mom is running on the execution nodes.

  • Check that the PBS server node is listed in the mom_priv/config file.

  • Check that the queues are enabled on the server.

Cannot Submit Jobs for PBS as root

If the root user is prevented from submitting jobs, use another user ID. The default installed SGI GenomeCluster PBS will not be configured so that root can submit requests.

To configure PBS to allow root to submit requests, execute the following on the head node:

# /usr/local/bin/qmgr -c 'set server acl_roots="root@*" '

Getting PBS State Information

To get PBS state information on all nodes, execute the following on the head node:

# /usr/local/bin/pbsnodes -a

If a node is missing, add it to the /usr/spool/pbs/server_priv/nodes file and restart PBS by entering the following on the head node:

# /etc/rc.d/init.d/pbs restart

If a node is down, ensure that the execution node's /usr/spool/pbs/server_name file contains the name of the head node. You must restart pbs_mom by entering the following on the execution node:

# /etc/rc.d/init.d/pbs_mom restart

It may take a few moments for the head node to synchronize up with the execution node.

For more information, see the PBS Administrator's Guide.

Root Access for LSF

By default, root access across the cluster is not allowed. To allow root access from a remote cluster, specify LSF_ROOT_REX=all in the lsf.conf file. This specification implies that root jobs from both the local and remote clusters are accepted. This applies to both interactive and batch jobs.

If you want clusters clus1 and clus2 to allow root access executions for local jobs only, you must specify LSF_ROOT_REX=local in the lsf.conf files of both clus1 and clus2. However, if you want clus2 to also allow root access execution from any cluster, change the line in the lsf.conf file for clus2 to LSF_ROOT_REX=all.


Note: The lsf.conf file is host-type specific and not shared across different platforms. You must make sure that the lsf.conf file for all of your host types are changed consistenly.


PCP Errors

PCP applications will return the following error if the PCP daemons have quit or have not previously been started:

application: hinv.ncpu: Unknown metric name
application: Failed to get the number of CPUs from host "test"

To resolve this problem, restart the PCP daemons as follows:

# /etc/rc.d/init.d/pcp stop
# /etc/rc.d/init.d/pcp start

The PCP daemons must be running on every execution node in the cluster (optionally on the head node).

For information about other problems, see the following files:

  • /usr/doc/pcp-version/README

  • /usr/doc/pcp-ace-version/README

  • /usr/doc/pcp-pro-version/README

Queue Priorities/Nice and PCP

PCP does not measure the nice priority. The LSF default queue has a nice priority of 20. You should change the queue nice priority to something that will not be preempted and that can be monitored. For example, the contents of the /usr/local/lsf/mnt/conf/lsbatch/bio/configdir/lsb.queues file should be similar to the following:

Begin Queue
QUEUE_NAME = normal
PRIORITY  = 1
NICE = 0
MEMLIMIT = 5000  # jobs bigger than this (5M) will be niced
STACKLIMIT = 2048
DESCRIPTION = For normal low priority jobs, running only if hosts are lightly loaded.
#QJOB_LIMIT = 60  # job limit of the queue
#UJOB_LIMIT = 5   # job limit per user
#PJOB_LIMIT = 2   # job limit per processor
#RUN_WINDOW = 5:19:00-1:8:30 20:00-8:30
#r1m = 0.7/2.0    # loadSched/loadStop
#r15m = 1.0/2.5
#pg = 4.0/8
#ut = 0.2
#io = 50/240
#CPULIMIT = 180/apple  # 3 hours of host apple
#FILELIMIT = 20000
#DATALIMIT = 20000     # jobs data segment limit
#CORELIMIT = 20000
#PROCLIMIT = 5         # job processor limit
#USERS = all           # users who can submit jobs to this queue
#HOSTS = all           # hosts on which jobs in this queue can run
#PRE_EXEC = /usr/local/lsf/misc/testq_pre >> /tmp/pre.out
#POST_EXEC = /usr/local/lsf/misc/testq_post |grep -v "Hey"
#REQUEUE_EXIT_VALUES = 55 34 78
End Queue  

NFS Version Defaults to 2

The nfs(5) man page incorrectly says that the NFS version defaults to version 2. This is a problem when integrating NFS into a standard Linux environment. To get around this problem, you must mount the other Linux filesystems as NFS version 2 by using nfsvers=2 in the /etc/fstab file.

For example:

voila:/users  /users nfs     exec,dev,suid,rw,nfsvers=2 1 1

For more information, see the fstab(5) man page.