This chapter describes how to administrate the interoperability of NQE tasks and Distributed Computing Environment (DCE) resources such as the Distributed File System (DFS).
 | Note:: DCE support is optional, and is disabled by default. It is a licensed feature of NQE. |
The following topics are discussed in this chapter:
Enabling DCE support for NQE
Using a password to submit a task
Using a forwardable ticket to submit a task
Checkpoint and restart support
Support for renewing and refreshing tickets
User requirements
Administrating Kerberos and DCE
Before enabling DCE support, system administrators should note the following:
The NQE Distributed Computing Environment/Distributed File Service (DCE/DFS) feature is restricted to operating within a single DCE cell. Cross realm authentication is not supported.
Ticket forwarding is dependent upon the use of the NQE database when submitting tasks.
Support for tasks that use forwarded tickets for DCE authentication is provided on UNICOS, UNICOS/mk, IRIX, and Solaris platforms only. Support for tasks that use a password for DCE authentication is available on all NQE 3.3 (or later) platforms. Tickets may be forwarded from any NQE 3.3 or later client to any NQE 3.3 (or later) server that supports forwarded tickets as a means of DCE authentication. (Ticket forwarding for either clients or servers is not supported for the Digital UNIX platform in the NQE 3.3 release.)
NQE support for DCE/DFS does not include installations residing in DFS accessible file space. Installed components must reside in UNIX file space.
If DCE support is enabled, a system cannot be configured to disable ticket forwarding.
Kerberos is the authentication component of DCE.
NQE supports Open Software Foundation (OSF) DCE version 1.1 only.
Note: For UNICOS and IRIX systems, the DCE integrated login feature must be enabled for DCE credentials to be passed through NQE. For more information, see the Cray DCE Client Services/Cray DCE DFS Server Release Overview, publication RO-5225.
To enable DCE support on an NQE node, use the Action menu of the NQE configuration utility, nqeconfig(8)), to add the following variables to the nqeinfo file:
NQE_AUTHENTICATION. Set the NQE authentication type. The value of this variable should be dce to enable DCE/DFS support.
NQE_DCE_REFRESH. If DCE support is enabled, DCE renew/refresh support is automatically set to a value of 300 minutes. The value assigned to this variable must be an integer representing the number of minutes between each renew/refresh attempt. The value should be approximately half of the minimum ticket lifetime for any user. A value of 0 disables the DCE refresh feature.
NQE_DCE_BIN. If the path to the kdestroy and kinit commands is not the default /opt/dcelocal/bin path, then add the NQE_DCE_BIN variable to the nqeinfo file, and set it to the correct full path name.
To enable DCE ticket forwarding support on an NQE client machine, only the NQE_AUTHENTICATION variable needs to be set in the nqeinfo file on the client.
In order for a task, which utilizes DCE authentication, such as DFS, to access resources , it must first acquire credentials. This is accomplished in two ways; either by providing a password or by forwarding existing credentials. When a user submits a task to NQE, they may choose to provide a password. The password remains with the task until it is complete, and it is used by NQS to acquire and refresh credentials for the user on the execution host. When submitting a task from the command line, a user may provide a password by appending the -P argument to the cqsub(1) command. The user will be prompted for their password prior to submission of the task. When submitting a task using the NQE GUI, a user may specify a password by selecting the Set Password item of the Actions menu in the NQE Job Submission window.
| Note: This action is separate from the user authentication performed by NQS when queuing a remotely submitted request or validating an alternate user identity. Although a password is required to obtain DCE credentials, the NQS user validation type defined within qmgr can remain unchanged. |
 | Caution: If user home directories are located in DFS space, NQS user validation must be set to password or no_validation if ticket forwarding will not be used . |
The integrated login feature is used on UNICOS or IRIX NQE servers for DCE authentication when a password is supplied. The DCE credentials acquired through integrated login are used for both initiating the request and returning request output files. Because integrated login is unavailable on most UNIX NQE servers, NQS makes DCE security login library calls to obtain separate sets of DCE credentials when initiating the request and returning request output files.
Failure to obtain DCE credentials results in a nonfatal error on all NQE platforms. The request will be initiated even if the attempt to obtain DCE credentials for the request owner fails. If DCE credentials are successfully obtained, the KRB5CCNAME environment variable is set within the request process that is initiated.
| Note: Users may use the klist(1) command within a request script to verify that DCE credentials were obtained. |
Messages are written to the NQS log file when attempts are made to obtain DCE credentials. A sample message indicating a successful attempt to obtain DCE credentials follows:
Request 1927.latte: Acquired DCE credentials for user <jane>, KRB5CCNAME <FILE:/opt/dcelocal/var/security/creds/dcecred_41fffce1>. |
A sample message indicating an unsuccessful attempt to obtain DCE credentials follows:
Request 1928.latte: Failed to get DCE credentials for user <jane>. |
Another method of DCE authentication involves forwarding a user's credentials between machines to submit a task. When a user obtains their initial DCE credentials using the kinit(1) command, they may explicitly request that they be granted a forwardable ticket. This allows the user to establish a new DCE context without having to provide their password again. After the user provides their password to the kinit command, their credential cache file is populated with a forwardable Kerberos ticket.
A user may also request that their DCE credentials be renewable. Because credentials have lifetimes associated with them, they eventually expire. When a user's credentials expire, they are no longer granted access to resources that utilize DCE as an authentication mechanism. The user's credentials have two lifetimes associated with them; the ticket lifetime and the renewable lifetime. The ticket lifetime is usually less than one day. The renewable lifetime is generally much longer than the ticket lifetime, and may last for weeks or months. Before the ticket lifetime of a user's credentials expires, they may request that it be renewed. If the ticket has not exceeded its maximum renewable lifetime, the lifetime of the ticket is reset. In this way, a user's credentials may be valid throughout the maximum renewable lifetime of the ticket.
For information on DCE policy regarding ticket lifetime, see the OSF DCE Administration Guide - Core Services.
In order to use ticket forwarding, DCE authentication must be enabled for NQE (see “Enabling DCE Support for NQE”). Prior to submitting a task to NQE, a user must acquire a forwardable and renewable Kerberos ticket granting ticket (TGT). This may be accomplished by using the kinit(1) command. Here is an example:
/opt/dcelocal/bin/kinit -f -r 100h jane |
This command specifies that user jane is requesting that a forwardable ticket be granted with a renewable lifetime of one hundred hours. The user is then prompted for their DCE password. Upon entering the correct password, a credential cache file may be created if one does not already exist. Otherwise, the user's existing credential cache will be updated.
| Note: Although this example refers specifically to the kinit(1) command supplied with DCE, a properly configured version of kinit from the Massachusetts Institute of Technology (MIT) Kerberos Version 5 package may also be used. In this way, it is possible for an NQE client machine without DCE installed to submit tasks that access DCE dependent resources when they are executed. |
The user may now use the cqsub(1) command to submit a task to the NQE database. It is not necessary to pass any additional arguments to the cqsub command to enable ticket forwarding. Forwarding is attempted by default when NQE authentication is set to DCE. If the user's credentials could not be forwarded, an informational message will be printed as shown in the following example:
CQSUB: INFO: Unable to forward Kerberos ticket: <Explanation> |
This is not a fatal error. The task will be scheduled to run, although it will not obtain DCE credentials unless an appropriate password was provided with the task.
On UNICOS systems, requests accessing DFS files can be checkpointed and restarted. NQS obtains new DCE credentials for the request owner just before restarting the request. The qmgr(8) subcommands which checkpoint and restart requests, handle this automatically.
| Caution: A restarted job correctly gets the new credentials obtained from NQS, but the KRB5CCNAME environment variable within the restart file is not reset to the new cache file name. After the job is restarted, a klist within the job script will incorrectly state that there are no credentials. Consequently, DCE services are affected but not DFS, which continues to work with the new credentials. |
In order for a user's credentials to remain valid throughout the life of a task, it may be necessary to periodically refresh the credentials . The DCE credential renew/refresh feature within NQE provides this functionality. For tasks using DCE ticket forwarding, it allows the user's credentials to remain valid throughout the maximum renewable lifetime of the user's initial ticket granting ticket. For tasks submitted with a password, the ticket may be refreshed as long as the password remains valid for the user. This feature is available on all NQE servers that support DCE.
When a task is submitted to the NQE database, the scheduler may choose not to assign the task to a lightweight server (LWS) immediately. The scheduler provided with the NQE 3.3 release periodically refreshes the credentials associated with the task if DCE ticket forwarding is being used.
| Note: It will be necessary to make changes to your default scheduler if you are upgrading to the NQE 3.3 release, and plan to use your old scheduler with this feature. Pay special attention to the addition of the ticket commands within the new scheduler.
Once the task has been scheduled, the LWS will submit the job to its NQS server. The NQS request server will acquire credentials for the task and continue to renew/refresh the credentials until the task is complete. |
The default renew/refresh interval is 300 minutes. This value should be approximately half of the minimum ticket lifetime for any user. For example, if user jane has the smallest default ticket lifetime of any user configured in the DCE cell of two hours, then the refresh interval should be configured to a value of 60. To change the renew/refresh interval, see “Enabling DCE Support for NQE”.
Users who submit tasks that make use of NQE DCE/DFS interoperability should be aware of the following:
Tasks must be submitted to the NQE database. If this is not configured as the default behavior, the user must include -d nqedb as part of the cqsub(1) command arguments. This behavior must be configured in the NQE Job Submission Options window when using the NQE GUI to submit tasks.
A password, forwardable credentials, or both must be available at task submission time in order for credentials to be acquired when the task is executed.
A user's home directory may reside within DFS space.
The request script file may reside within DFS space.
The target directory for job output may reside within DFS space. Directing output to DFS space may be accomplished by using the standard cqsub arguments, which specify the destination of standard output (-o) and standard error (-e) files. The format of the DFS path should begin with /:/ followed by the remainder of the desired location.
The user supplied password and the DCE registry password for the user must be identical.
Caution: NQS limits a password to eight characters. If a user's DCE password is more than eight characters, DCE authentication will fail and credentials will not be acquired.
A user may provide a different DCE registry user name when submitting a request by using the User Name field of the NQE GUI Submit window General Options menu or by using the -u option of the cqsub(1) command.
After a request completes, NQS on UNICOS systems uses the kdestroy(1) command to destroy any credentials obtained by NQS on behalf of the request owner. On non-UNICOS systems, this is accomplished by calling built-in Kerberos library routines.
| Caution: Including a kdestroy command within a request script file on UNICOS systems will destroy the credentials obtained by NQS and prevent NQS from returning request output files into DFS space. |
In order for NQE tasks to utilize DCE/DFS resources, it is necessary to properly configure your DCE/DFS environment. To accomplish this, several issues must be addressed. These issues can be broken into two main groups; Kerberos specific and DCE/DFS specific.
Configuration issues relating specifically to DCE are as follows:
User accounts (or principals) within DCE should be inspected to determine the range of ticket lifetimes. As discussed earlier, the renew/refresh interval within NQE should be approximately half of the minimum configured value in this range. Performing a ticket refresh is not a particularly expensive operation in most cases, but need not be performed at an unnecessarily high frequency. The administrator may choose to extend the ticket lifetime of all user principals to a period of at least ten hours. This allows the default NQE renew/refresh interval (300 minutes = 5 hours) to fit appropriately with the ticket lifetimes of the user principals.
User accounts (or principals) within DCE should be inspected to determine the range of renewable lifetimes. The renewable lifetime of a principal should always exceed the maximum projected task lifetime (time from task submission to completion) for a user. If a task lives longer than the renewable lifetime of its associated credentials, the task will lose access to DCE/DFS resources unless a valid DCE password is associated with the task. This is most important for tasks utilizing DCE ticket forwarding.
User accounts (or principals) within DCE should be set to allow forwardable, renewable, and proxiable certificates.
The account and password lifespan policies should be inspected. If an account expires, or if its password becomes invalid, the task may lose access to DCE controlled resources.
Kerberos performs authentication in part based on an IP address encrypted within the ticket. The KDC (or DCE security service) checks that the ticket granting ticket (TGT) contains a IP address that matches the IP address of the interface that the request was made on. This can create difficulty for machines that have multiple IP addresses (multi-homed). The Domain Name Service (DNS) can store multiple interface IP addresses for a multi-homed host. Therefore , if you run named(8) on your Cray Research system you can accommodate multi-homed hosts. Kerberos 5 has the ability to store multiple IP addresses for a host within the TGT. If you do not run named you must ensure that the IP address provided by a gethostbyname(3c) call will return the same address on every machine that is to act as a login utility client (that is the host you wish to run klogin on for example). This IP address must also be the interface that the destination host (the host running klogind) will use to make requests to the KDC (DCE security service).
Configuration issues relating specifically to Kerberos are as follows:
All systems that have both NQE and DCE or Kerberos installed will need to be configured for Kerberos.
The file /etc/krb5.conf should contain realm specific data. In the following example, the default realm is configured to be the system latte, which acts as a DCE key distribution center (KDC):
[libdefaults] default_realm = latte.cray.com default_tkt_enctypes = des-cbc-crc default_tgs_enctypes = des-cbc-crc kdc_req_checksum_type = 2 ccache_type = 2 [realms] latte.cray.com = { kdc = latte default_domain = cray.com } [domain_realm] .cray.com = latte.cray.com
The entries in /etc/services must be configured so that either the kerberos or kerberos-sec services operate on port 88/udp. To preserve an existing Kerberos V4 configuration, use the following entry:
kerberos-sec 88/udp kdc
For Kerberos V5 only, the entry may be as follows:
kerberos 88/udp kdc
For more detailed information regarding Kerberos configuration, please refer to the Kerberos V5 Installation Guide available from MIT.