Sun Grid Engine Information Center
Index

Administering Sun Grid Engine

As a Sun Grid Engine administrator, you need to perform the following tasks:

• Add or remove Grid Engine hosts
• Configure hosts and queues
• Manage user access
• Monitor Sun Grid Engine to ensure it is functioning as you need it to function
• Perform general maintenance tasks
• Perform special tasks, such as setting up parallel jobs

Administration Tools

To perform these tasks, you can use either of the following mechanisms:

• A graphical user interface, QMON
• A command-line interface, mainly centered on the use of the following commands:
  • qconf — Add, delete, and modify the current Grid Engine configuration
  • qhost — View current status of the available Grid Engine hosts, the queues, and the jobs associated with the queues
  • qalter and qsub — Submit jobs
  • qstat — Show the status of Grid Engine jobs and queues
  • qquota — List each resource quota that is being used at least once or that defines a static limit

For general information about these administration tools, see Interacting With Sun Grid Engine.

Administration Tasks

For detailed information about performing Grid Engine administration tasks, see:

• Printable Administering Sun Grid Engine Guide
• Managing User Access
  • Setting Up a User
  • Configuring User Access
  • Defining Projects
  • Configuring Default Requests
  • Using Path Aliasing
• Configuring Hosts and Clusters
  • Basic Cluster Configuration
  • Changing the Master Host
  • Configuring Shadow Master Hosts
  • Configuring Hosts With QMON
  • Configuring Hosts From the Command Line
• Configuring Queues
• Configuring Queue Calendars
• Managing the Scheduler
• Managing Policies
• Managing Resource Quotas
• Managing Advance Reservations
• Managing Parallel Environments
• Managing Checkpointing Environments
• Configuring Complex Resource Attributes
• Load Parameters
• Managing Grid Engine SMF Services
• Generating Accounting Statistics
• Backing Up and Restoring Grid Engine Configuration
• Improving Grid Engine Performance
• Using Files and Scripts for Administrative Tasks

To print this section, see the Printable Administering Sun Grid Engine Guide.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Managing User Access

This section contains information about managing user accounts and other related accounts. The following topics are covered:

• Setting Up a User
• Configuring User Access
• Defining Projects
• Configuring Default Requests
• Using Path Aliasing

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Setting Up a User

You need to perform the following tasks to set up a user for the Grid Engine system:

• Assign required logins.
  To submit jobs from host A for execution on host B, users must have identical accounts on both hosts. No login is required on the machine where sge_qmaster runs.

• Set access permissions.
  The Grid Engine software enables you to restrict user access to the entire cluster, to queues, and to parallel environments. See Configuring User Access for a detailed description.
  In addition, you can grant users permission to suspend or enable certain queues. See How to Configure User Access Parameters for more information.

• Declare a Grid Engine System user.
  To add users to the share tree or to define functional or override policies for users, you must declare those users to the Grid Engine system. For more information, see Configuring Policy-Based Resource Management With QMON and Configuring User Objects With QMON.

• Set up project access.
  If projects are used for the definition of share-based, functional, or override policies, you should give the user access to one or more projects. Otherwise the user's jobs might end up in the lowest possible priority class, which would result in the jobs having access to very few resources. See Configuring Policy-Based Resource Management With QMON for more information.

• Set file access restrictions.
  Users of the Grid Engine system must have read access to the directory $SGE_ROOT/$SGE_CELL/common. Before a job starts, the execution daemon creates a temporary working directory for the job and changes ownership of the directory to the job owner. The execution daemon runs as root. The temporary directory is removed as soon as the job finishes. The temporary working directory is created under the path defined by the queue configuration parameter tmpdir. See the queue_conf(5) man page for more information.
  Make sure that temporary directories can be created under the tmpdir location. The directories should be set to Grid Engine system user ownership. Users should be able to write to the temporary directories.

• Set up site dependencies.
  By definition, batch jobs do not have a terminal connection. Therefore UNIX commands like stty in the command interpreter's startup resource file (for example, .cshrc for csh) can lead to errors. Check for the occurrence of stty in startup files. Avoid the commands that are described in Verifying the Installation.
  Because batch jobs are usually run off line, the Grid Engine system notifies users of error events in the following ways:
  • Logs error messages to the Grid Engine system log file.
  • Sends an email to the job owner. If the error log file can't be opened, email is the only way to notify the user of an error event. Therefore, the email system should be properly installed for Grid Engine users.

• Set up Grid Engine system definition files.
  You can set up the following definition files for Grid Engine users:
  • qmon – Resource file for the Grid Engine system GUI. See Customizing QMON.
  • sge_aliases – Aliases for the path to the current working directory. See Using Path Aliasing.
  • sge_request – Default request definition file. See Configuring Default Requests.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring User Access

Types of Users

The Grid Engine system has the following four categories of users:

• Managers – Managers have full capabilities to manipulate the Grid Engine system. By default, the superusers of the master host and of any machine that hosts a queue instance have manager privileges.
• Operators – Operators can perform many of the same commands as managers, except that operators cannot add, delete, or modify queues.
• Users – Users have certain access permissions, as described in Configuring Users, but users have no cluster or queue management capabilities.

Queue owners can be managers, operators, or users. Queue owners are restricted to suspending and resuming, or disabling and enabling, the queues that they own. These privileges are necessary for successful use of qidle. Users are commonly declared to be owners of the queue instances that reside on their desktop workstations. See How to Configure Owners Parameters for more information.

Configuring Manager Accounts

How to Configure Manager Accounts With QMON

1. On the QMON Main Control window, click the User Configuration button.
   The Manager tab appears, shown in the following figure, and lists all accounts that have administrative permission.
   
   
   
2. To enable another user to manage the Grid Engine system, type the user name in the field above the manager account list.
   Click Add or press the Return key.
   
   
3. To delete a manager account, select it, and then click Delete.

Configuring Manager Accounts From the Command Line

To configure a manager account from the command line, type the following command with appropriate options:

# qconf <options>

The following options are available:

• qconf -am username – The -am option (add manager) adds one or more users to the list of Grid Engine system managers. By default, the root accounts of all trusted hosts are Grid Engine system managers. See About Hosts and Daemons for more information.
• qconf -dm username – The -dm option (delete manager) deletes the specified users from the list of Grid Engine system managers.
• qconf -sm – The -sm option (show managers) displays a list of all Grid Engine system managers.

Configuring Operator Accounts

How to Configure Operator Accounts With QMON

1. On the QMON Main Control window, click the User Configuration button, and then click the Operator tab.
   The Operator tab, similar to the Manager tab shown in the figure above, and lists all accounts that currently have restricted administrative permission. If the account also has manager access, then that overrides operator access. See Configuring Manager Accounts With QMON.
   
   
2. To add a new operator account, type its name in the field above the operator account list.
   Click Add or press the Return key.
   
   
3. To delete an operator account, select it, and then click Delete.

Configuring Operator Accounts From the Command Line

To configure an operator account from the command line, type the following command with appropriate options:

# qconf <options>

The following options are available:

• qconf -ao username – The -ao option (add operator) adds one or more users to the list of Grid Engine system operators.
• qconf -do username – The -do option (delete operator) deletes the specified users from the list of Grid Engine system operators.
• qconf -so – The -so option (show operators) displays a list of all Grid Engine system operators.

Configuring User Access Lists

Any user with a valid login ID on at least one submit host and one execution host can use the Grid Engine system. However, Grid Engine system managers can prohibit access for certain users to certain queues or to all queues. Furthermore, managers can restrict the use of facilities such as specific parallel environments. See Configuring Parallel Environments for more information.

To define access permissions, you must define user access lists, which are made up of named sets of users. In the Grid Engine system, these are referred to as usersets. You use user names and UNIX group names to define user access lists. The user access lists are then used either to deny or to allow access to a specific resource in any of the following configurations:

• Cluster configuration – see Basic Cluster Configuration.
• Queue configuration – see Configuring Subordinate Queues.
• Parallel environment configuration – see Configuring Parallel Environments With QMON.

Usersets are also used to define Grid Engine system projects and departments. For details about projects, see Defining Projects.

How to Configure User Access Lists With QMON

1. On the QMON Main Control window, click the User Configuration button, and then click the Userset tab.
   The Userset tab appears as shown in the following figure.
   
   
   In the Grid Engine system, a userset can be either an access list, a Department, or both. The check boxes below the Usersets list indicate the type of the selected userset. This section describes access lists. Departments are explained in Defining Usersets As Projects and Departments.
   
   The Usersets list displays all available access lists. To display currently defined users and groups, select the userset.

   Note The names of groups are prefixed with an @ sign.

   
   

2. To add, modify, or delete a userset, do the following:
   • To add a new userset, click Add.
     An Access List Definition dialog box appears, as shown in the figure below. The Users/Groups list displays all currently defined users and groups.
     
     • To add a new access list definition, type the name of the access list in the Userset Name field and click Ok.
     • To add a new user or group to the access list, type a user or group name in the User/Group field and then click Ok. Be sure to prefix group names with an @ sign.
     • To delete a user or group from the Users/Groups list, select it and then click the trash icon.
   • To modify an existing userset, select it, and then click Modify.
     An Access List Definition dialog box appears with the name of the current userset in the Userset Name field.
   • To delete a userset, select it, and then click Delete.
     
3. To save your changes and close the dialog box, click OK.
   Click Cancel to close the dialog box without saving changes.
   
   
4. To close the User Configuration dialog box, click Done.

Configuring User Access Lists From the Command Line

To configure user access lists from the command line, type the following command with appropriate options:

# qconf <options>

The following options are available:

• qconf -au username [,...] access-list-name [,...] – The -au option (add user) adds one or more users to the specified access lists.
• qconf -Au filename – The -Au option (add user access list from file) uses a configuration file, filename, to add an access list.
• qconf -du username [,...] access-list-name [,...] – The -du option (delete user) deletes one or more users from the specified access lists.
• qconf -dul access-list-name [,...] – The -dul option (delete user list) completely removes userset lists.
• qconf -mu access-list-name – The -mu option (modify user access list) modifies the specified access lists.
• qconf -Mu filename – The -Mu option (modify user access list from file) uses a configuration file, filename, to modify the specified access lists.
• qconf -su access-list-name [,...] – The -su option (show user access list) displays the specified access lists.
• qconf -sul – The -sul option (show user access lists) displays all access lists currently defined.

Configuring Users

You must declare user names before you define the share-based, functional, or override policies for users. See Configuring Policy-Based Resource Management With QMON.

If you do not want to explicitly declare user names before you define policies, the Grid Engine system can automatically create users for you, based on predefined default values. The automatic creation of users can significantly reduce the administrative burden for sites with many users.

To have the system create users automatically, set the Enforce User parameter on the Cluster Settings dialog box to Auto. To set default values for automatically created users, specify values for the following Automatic User Defaults on the Cluster Settings dialog box:

• Override Tickets
• Functional Shares
• Default Project
• Delete Time

For more information about the cluster configuration, see Basic Cluster Configuration.

How to Configure User Objects With QMON

1. On the QMON Main Control window, click the User Configuration button.
   
   
2. Click the User tab.
   The User tab appears as shown in the following figure:
   
   
   
3. To add a new user, type a user name in the field above the User list, and then click Add or press the Return key.
   To delete a user, select the user name in the User list, and then click Delete.
   The Delete Time column is read-only. The column indicates the time at which automatically created users are to be deleted from the Grid Engine system. Zero indicates that the user will never be deleted.
   
   
4. To assign a default project, select a user, and then click the Default Project column heading.
   A Project Selection dialog box appears, as shown below. You can assign a default project to each user. The default project is attached to each job that users submit, unless those users request another project to which they have access.
   
   Departments are used for the configuration of the functional policy and the override policy. Departments differ from access lists in that a user can be a member of only one department, whereas one user can be included in multiple access lists. For more details, see Configuring the Functional Policy and Configuring the Override Policy.
   
   A Userset is identified as a department by the Department flag. A Userset can be defined as both a department and an access list at the same time. However, the restriction of only a single appearance by any user in any department applies.
   
   
   
5. Select a project for the highlighted user entry.
   
   
6. Click OK to assign the default project and close the dialog box.
   Click Cancel to close the dialog box without assigning the default project.

Configuring User Objects From the Command Line

To configure user objects from the command line, type the following command with appropriate options:

# qconf <options>

The following options are available:

• qconf -auser – The -auser option (add user) opens a template user configuration in an editor. See the user(5) man page. The editor is either the default vi editor or the editor specified by the EDITOR environment variable. After you save your changes and exit the editor, the changes are registered with sge_qmaster.
• qconf -Auser filename – The -Auser option (add user from file) parses the specified file and adds the user configuration. The file must have the format of the user configuration template.
• qconf -duser username [,...] – The -duser option (delete user) deletes one or more user objects.
• qconf -muser username – The -muser option (modify user) enables you to modify an existing user entry. The option loads the user configuration in an editor. The editor is either the default vi editor or the editor specified by the EDITOR environment variable. After you save your changes and exit the editor, the changes are registered with sge_qmaster.
• qconf -Muser filename – The -Muser option (modify user from file) parses the specified file and modifies the user configuration. The file must have the format of the user configuration template.
• qconf -suser username – The -suser option (show user) displays the configuration of the specified user.
• qconf -suserl – The -suserl option (show user list) displays a list of all currently defined users.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Defining Projects

About Projects

Projects provide a means to organize joint computational tasks from multiple users. A project also defines resource usage policies for all jobs that belong to such a project.

Projects must be declared before they can be used in any of the three scheduling policy policies. Projects are used in three policy areas:

• Share-based, when shares are assigned to projects – see Configuring the Share-Based Policy
• Functional, when projects receive a percentage of the functional tickets – see Configuring the Functional Policy
• Override, when an administrator grants override tickets to a project – see Configuring the Override Policy

Grid Engine system managers define projects by giving them a name and some attributes. Grid Engine users can attach a job to a project when they submit the job. Attachment of a job to a project influences the job's dispatching, depending on the project's share of share-based, functional, or override tickets.

How to Define Projects With QMON

Grid Engine system managers can define and update definitions of projects by using the Project Configuration dialog box.

1. To define a project, on the QMON Main Control window, click the Project Configuration button.
   The Project Configuration dialog box appears, as shown below. The currently defined projects are displayed in the Projects list. The project definition of a selected project is displayed under Configuration.
   
   
   
2. To add a new project, click Add.
   
   
3. To modify a project, select it, and then click Modify.
   Clicking Add or Modify opens the Add/Modify Project dialog box.
   
   
   The name of the selected project is displayed in the Name field. The project defines the access lists of users who are permitted access or who are denied access to the project. See Configuring Users for more information.
   
   The Add/Modify Project dialog box shows the following access lists:
   • The User Lists column shows access lists that contain users who have permission to access the project.
   • The Xuser Lists column shows access lists that contain users who do not have permission to access the project.
   • If both lists are empty, all users can access the project. If a user belongs to an access list that is in the User Lists and to an access list that is in the Xuser Lists, the user is denied access to the project.
     
4. To delete a project immediately, select it, and then click Delete.
   
   
5. To add or remove users from the User Lists or Xuser Lists, click the button at the right of the User Lists or the Xuser Lists.
   The Select Access Lists dialog box, shown in the figure below, appears. Currently defined access lists are displayed under Available Access Lists. Currently selected access lists are displayed under chosen access lists. You can select access lists in either list. You can move access lists from one list to the other by using the red arrows.
   
   
   
6. Click OK to save your changes and close the dialog box.
   Click Cancel to close the dialog box without saving your changes.

Defining Projects From the Command Line

To define projects from the command line, type the following command with appropriate options:

# qconf <options>

The following options are available:

• qconf -aprj – The -aprj option (add project) opens a template project configuration in an editor. See the project(5) man page. The editor is either the default vi editor or the editor specified by the EDITOR environment variable. After you save your changes and exit the editor, the changes are registered with sge_qmaster.
• qconf -Aprj filename – The -Aprj option (add project from file) parses the specified file and adds the new project configuration. The file must have the format of the project configuration template.
• qconf -dprj project-name – The -dprj option (delete project) deletes one or more projects.
• qconf -mprj project-name – The -mprj option (modify project) enables you to modify an existing user entry. The option loads the project configuration in an editor. The editor is either the default vi editor or the editor specified by the EDITOR environment variable. After you save your changes and exit the editor, the changes are registered with sge_qmaster.
• qconf -Mprj filename – The -Mprj option (modify project from file) parses the specified file and modifies the existing project configuration. The file must have the format of the project configuration template.
• qconf -sprj project-name – The -sprj option (show project) displays the configuration of a particular project.
• qconf -sprjl – The -sprjl option (show project list) displays a list of all currently defined projects.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Default Requests

Batch jobs are normally assigned to queues with respect to a request profile. The user defines a request profile for a particular job. The user assembles a set of requests that must be met to successfully run the job. The scheduler considers only those queues that satisfy the set of requests for this job.

If the user does not specify any requests for a job, the scheduler considers any queue to which the user has access without further restrictions. However, the Grid Engine software enables you to configure default requests that define resource requirements for jobs even when the user does not specify resource requirements explicitly.

You can configure default requests globally for all users of a cluster, as well as privately for any user. The default request configuration is stored in default request files. The global request file is located under $SGE_ROOT/$SGE_CELL/common/sge_request. The user-specific request file can be located either in the user's home directory or in the current working directory. The working directory is where the qsub command is run. The user-specific request file is called .sge_request.

If these files are present, they are evaluated for every job. The order of evaluation is as follows:

1. The global default request file
2. The user default request file in the user's home directory
3. The user default request file in the current working directory

Note The requests specified in the job script or supplied with the qsub command take precedence over the requests in the default request files. See Submitting Jobs for details about how to request resources for jobs explicitly.

You can prevent the Grid Engine system from using the default request files by using the qsub -clear command, which discards any previous requirement specifications.

Format of Default Request Files

The format of both the local and the global default request files is as follows:

• Default request files can contain any number of lines. Blank lines and lines that begin with a # sign are skipped.
• Each line not to be skipped can contain any qsub option, as described in the qsub(1) man page. More than one option per line is allowed. The batch script file and the argument options to the batch script are not considered to be qsub options. Therefore these items are not allowed in a default request file.
• The qsub -clear command discards any previous requirement specifications in the currently evaluated request file or in request files processed earlier.

Suppose a user's local default request file is configured the same as test.sh, the script in the following example.

Example of a Default Request File

# Local Default Request File
# exec job on a sun4 queue offering 5h cpu
-l arch=solaris64,s_cpu=5:0:0
# exec job in current working dir
-cwd

To run the script, the user types the following command:

% qsub test.sh

The effect of running the test.sh script is the same as if the user specified all qsub options directly in the command line, as follows:

% qsub -l arch=solaris64,s_cpu=5:0:0 -cwd test.sh

Note Like batch jobs submitted using qsub, interactive jobs submitted using qsh consider default request files also. Interactive or batch jobs submitted using QMON also take these request files into account.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Using Path Aliasing

In Solaris and in other networked UNIX environments, users often have the same home directory, or part of it, on different machines. For example, consider user home directories that are available across NFS and automounter. A user might have a home directory /home/foo on the NFS server. This home directory is accessible under this path on all properly installed NFS clients that are running automounter. However, /home/foo on a client is just a symbolic link to /tmp_mnt/home/foo. /tmp_mnt/home/foo is the actual location on the NFS server from where automounter physically mounts the directory.

A user on a client host might use the qsub -cwd command to submit a job from somewhere within the home directory tree. The -cwd flag requires the job to be run in the current working directory. However, if the execution host is the NFS server, the Grid Engine system might not be able to locate the current working directory on that host. The reason is that the current working directory on the submit host is /tmp_mnt/home/foo, which is the physical location on the submit host. This path is passed to the execution host. However, if the execution host is the NFS server, the path cannot be resolved, because its physical home directory path is /home/foo, not /tmp_mnt/home/foo.

Other occasions that can cause similar problems are the following:

• Fixed NFS mounts with different mount point paths on different machines. An example is the mounting of home directories under /usr/people on one host and under /usr/users on another host.
• Symbolic links from outside into a network-available file system.

To prevent such problems, the Grid Engine software enables both the administrator and the user to configure a path aliasing file. The locations of two such files are as follows:

• $SGE_ROOT/$SGE_CELL/common/sge_aliases – A global cluster path-aliasing file for the cluster
• $HOME/.sge_aliases – A user-specific path-aliasing file

  Note Only an administrator should modify the global file.

Format of Path-Aliasing Files

Both path-aliasing files share the same format:

• Blank lines and lines that begin with a # sign are skipped.
• Each line, other than a blank line or a line preceded by #, must contain four strings separated by any number of blanks or tabs. The strings are as follows:
  • The first string specifies a source path.
  • The second string specifies a submit host.
  • The third string specifies an execution host.
  • The fourth string specifies the source path replacement.
• Both the submit host and the execution host strings can be an * (asterisk), which matches any host.

How Path-Aliasing Files Are Interpreted

The files are interpreted in the following order:

1. After qsub retrieves the physical current working directory path, the global path-aliasing file is read, if present.
   The user path-aliasing file is read afterwards, as if the user path-aliasing file were appended to the global file.
   
   
2. Lines not to be skipped are read from the top of the file, one by one. The translations specified by those lines are stored, if necessary.
   A translation is stored only if both of the following conditions are true:
   • The submit host string matches the host on which the qsub command is run.
   • The source path forms the initial part either of the current working directory or of the source path replacements already stored.
     
3. After both files are read, the stored path-aliasing information is passed to the execution host along with the submitted job.
   
   
4. On the execution host, the path-aliasing information is evaluated. The source path replacement replaces the leading part of the current working directory if the execution host string matches the execution host. In this case, the current working directory string is changed. To be applied, subsequent path aliases must match the replaced working directory path.
   
   The following is an example of how the NFS automounter problem described earlier can be resolved with an aliases file entry.

Example – Path Aliasing File

# cluster global path aliases file
# src-path  subm-host   exec-host   dest-path
/tmp_mnt/   *           *           /

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Hosts and Clusters

This section provides background information about configuring various aspects of the Grid Engine system.

For specific configuration tasks, see the following topics:

• Basic Cluster Configuration
• Changing the Master Host
• Configuring Shadow Master Hosts
• Configuring Hosts With QMON
• Configuring Hosts From the Command Line

About Hosts and Daemons

You can classify Grid Engine system hosts into four categories, depending on which daemons are running on the system and on how the hosts are registered at sge_qmaster:

Note A host can belong to more than one class. By default, the master host is an administration host and a submit host.

• Master host – The master host runs the master daemon sge_qmaster. sge_qmaster controls all Grid Engine system components such as queues and jobs. It also maintains tables that contain information such as user access permissions and the status of the components. The master host usually runs the scheduler sge_schedd. The master host requires no further configuration other than that performed by the installation procedure. For information about how to initially set up the master host, see How to Install the Master Host. For information about how to configure dynamic changes to the master host, see Configuring Shadow Master Hosts.
• Execution hosts – Execution hosts are nodes that have permission to run jobs. Therefore they host queue instances, and they run the execution daemon sge_execd. An execution host is initially set up by the installation procedure, as described in How to Install Execution Hosts.
• Administration hosts – Administration hosts have permission to perform any kind of administrative activity, even though they are not the master host. See the qconf(1) man page for details. Administrative hosts are set up with the following command:

  qconf -ah <hostname>
  

• Submit hosts – Submit hosts allow for submitting and controlling batch jobs only. In particular, a user who is logged into a submit host can use qsub to submit jobs, can use qstat to control the job status, or can run the graphical user interface QMON. Submit hosts are set up using the qconf -as hostname command. See the qconf(1) man page for details.

About Configuring Hosts

The Grid Engine software maintains object lists for all types of hosts except for the master host. The lists of administration host objects and submit host objects indicate whether a host has administrative or submit permission. The list of execution host objects includes other parameters. Among these parameters are the load information that is reported by the sge_execd running on the host, and the load parameter scaling factors that are defined by the administrator.

You can configure host objects with QMON or from the command line.

Invalid Host Names

The following host names are invalid, reserved, or otherwise not allowed to be used:

• global
• template
• all
• default
• unknown
• none

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Basic Cluster Configuration

About Basic Cluster Configuration

The basic cluster configuration is a set of information that is configured to reflect site dependencies and to influence Grid Engine system behavior. Site dependencies include valid paths for programs such as mail or xterm. A global configuration is provided for the master host as well as for every host in the Grid Engine system pool. In addition, you can configure the system to use a configuration local to each host to override particular entries in the global configuration.

The cluster administrator should adapt the global configuration and local host configurations to the site's needs immediately after the installation. The configurations should be kept up to date afterwards.

The sge_conf(5) man page contains a detailed description of the configuration entries.

Configuring Clusters With QMON

You use the QMON Main Control window to configure and view information about clusters.

How to Display Cluster Configuration With QMON

1. On the QMON Main Control window, click the Cluster Configuration button.
   The Cluster Configuration dialog box appears.
   
   
2. In the Host list, select the name of a host.
   The current configuration for the selected host is displayed under Configuration.

How to Display Global Cluster Configuration With QMON

1. On the QMON Main Control window, click the Cluster Configuration button.
   The Cluster Configuration dialog box appears.
   
   
2. In the Host list, select global.
   The configuration is displayed in the format that is described in the sge_conf(5) man page.

How to Add and Modify Global and Host Configurations With QMON

1. On the QMON Main Control window, click the Cluster Configuration button.
   The Cluster Configuration dialog box appears.
   
   
2. Select a host name or the name global, and then click the Add or Modify button.
   The Cluster Settings dialog box appears, as shown in the following figure.
   
   
   The Cluster Settings dialog box enables you to change all parameters of a global configuration or a local host configuration.
   • All fields of the dialog box are accessible only if you are modifying the global configuration.
   • If you modify a local host, its configuration is reflected in the dialog box. You can modify only those parameters that are feasible for local host changes.
   • If you are adding a new local host configuration, the dialog box fields are empty.
   • The Advanced Settings tab shows a corresponding behavior, depending on whether you are modifying a configuration or are adding a new configuration. The Advanced Settings tab provides access to more rarely used cluster configuration parameters, as shown in the following figure.
     
     
     
3. When you finish making changes, click the OK button to save your changes and close the dialog box.
   Click the Cancel button to close the dialog box without saving changes.

See the sge_conf(5) man page for a complete description of all cluster configuration parameters.

How to Delete a Cluster Configuration With QMON

1. On the QMON Main Control window, click the Cluster Configuration button.
   The Cluster Configuration dialog box appears.
   
   
2. In the Host list, select the name of a host whose configuration you want to delete, and then click Delete.

Working With Basic Cluster Configurations From the Command Line

You can display or modify cluster configurations from the command line.

Displaying Cluster Configurations From the Command Line

To display the current cluster configuration, use the qconf -sconf command. See the qconf(1) man page for a detailed description.

Type one of the following commands:

% qconf -sconf
% qconf -sconf global
% qconf -sconf <host>

• The qconf -sconf and qconf -sconf global commands are equivalent. They display the global configuration.
• The qconf -sconf host command displays the specified local host's configuration.

Modifying Cluster Configurations From the Command Line

Note You must be an administrator to use the qconf command to change cluster configurations.

Type one of the following commands:

% qconf -mconf global
% qconf -mconf <host>

• The qconf -mconf global command modifies the global configuration.
• The qconf -mconf host command modifies the local configuration of the specified execution host or master host.

The qconf commands that are described here are examples of the many available qconf commands. See the qconf(1) man page for others.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Changing the Master Host

This section explains how to change the system that Grid Engine considers to be the master host by moving the sge_qmaster daemon.

How to Migrate qmaster to Another Host by Using a Script

Note Because the spooling database cannot be located on an NFS-mounted file system, the following procedure requires that you use the Berkeley DB RPC server for spooling. If you configure spooling to a local file system, you must transfer the spooling database to a local file system on the new sge_qmaster host.

1. Check that the new master host has read/write access.
   The new master host must have read/write access to the qmaster spool directory and common directory as does the current master. If the administrative user is the root user (check the global cluster configuration for the setting of admin_user), you should verify that the root user can create files in these directory under the root user name.
   
   
2. Run the migration script on the new master host.
   On the new master host, type the following command as the root user:
   

   # $SGE_ROOT/$SGE_CELL/common/sgemaster -migrate
   

   This command stops sge_qmaster on the old master host and starts it on the new master host. The master host name listed in the file $SGE_ROOT/$SGE_CELL/common/act_qmaster is automatically changed to the new master host. If qmaster is not running, warning messages will appear and a delay of about one minute will occur until qmaster is started on the new host.
   
   

3. Modify the shadow_masters file if necessary.
   
   1. Check if the $SGE_ROOT/$SGE_CELL/common/shadow_masters file exists.
      If the file exists, you can add the new qmaster host to this file and remove the old master host, depending on your requirements.
   2. Then stop and restart the sge_shadowd daemons by issuing the following commands on the respective machines:
      

      $SGE_ROOT/$SGE_CELL/common/sgemaster -shadowd stop
      $SGE_ROOT/$SGE_CELL/common/sgemaster -shadowd start
      

Important Notes About Migration

The migration procedure migrates to the host on which the sgemaster -migrate command is issued. If the file primary_qmaster exists, any subsequent calls of sgemaster on the machine contained in the primary_qmaster file will cause a migration back to that machine. To avoid such a situation, change or delete the $SGE_ROOT/$SGE_CELL/common/primary_qmaster file.

Note Existence of the primary_qmaster file does not imply that the qmaster is actually running.

Although jobs may continue to run during the migration procedure, the grid should be inactive. While the migration is taking place, any running Grid Engine commands, such as qsub or qstat, will return an error.

If the current qmaster is down, the scheduler will not shut down until it times out waiting for contact with the qmaster.

The shadow_masters file has no direct effect on the migration procedure. This file only exists if one or more shadow masters have been configured. For more information on how to set up shadow masters, see Configuring Shadow Master Hosts.

How to Migrate qmaster to Another Host Manually

1. On the current master host, stop the master daemon.
   Type the following command:
   

   qconf -km
   

   
   

2. Edit the $SGE_ROOT/$SGE_CELL/common/act_qmaster file according to the following guidelines:
   • Confirm the new master host's name. To get the new master host name, type the following command on the new master host:
     

     $SGE_ROOT/utilbin/$SGE_ARCH/gethostname
     

   • In the act_qmaster file, replace the current host name with the new master host's name returned by the gethostname utility.
     
     
3. On the new master host, start sge_qmaster:
   

   $SGE_ROOT/$SGE_CELL/common/sgemaster
   

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Shadow Master Hosts

About Shadow Master Hosts

Shadow master hosts are machines in the cluster that can detect a failure of the master daemon and take over its role as master host. When the shadow master daemon detects that the master daemon sge_qmaster has failed abnormally, it starts up a new sge_qmaster daemon on the host where the shadow master daemon is running.

Note If the master daemon is shut down gracefully, the shadow master daemon does not start up. If you want the shadow master daemon to take over after you shut down the master daemon gracefully, remove the lock file that is located in the sge_qmaster spool directory. The default location of this spool directory is $SGE_ROOT/$SGE_CELL/spool/qmaster.

The automatic failover start of a sge_qmaster on a shadow master host takes approximately one minute. Meanwhile, you get an error message whenever a Grid Engine system command is run.

Note The file $SGE_ROOT/$SGE_CELL/common/act_qmaster contains the name of the host that is actually running the sge_qmaster daemon.

Shadow Master Host Requirements

To prepare a host as a shadow master, the following requirements must be met:

• The shadow master host must run sge_shadowd.
• The shadow master host must share sge_qmaster status information, job configuration, and queue configuration that resides in a log file. In particular, a shadow master host needs read/write root access to the master host's spool directory and to the directory $SGE_ROOT/$SGE_CELL/common.
• Either the Berkeley DB RPC server or classic Grid Engine system spooling must be used for sge_qmaster spooling. For more information, see Database Server and Spooling Host.
• The shadow master host file must contain a line that defines the host as shadow master host.

As soon as these requirements are met, the shadow-master-host facility is activated for this host. You do not have to restart the Grid Engine system daemons to activate the feature.

Shadow Master Host File

The shadow master host file, $SGE_ROOT/$SGE_CELL/common/shadow_masters, contains the following:

• The name of the primary master host, which is the machine where the master daemon sge_qmaster initially runs
• The names of the shadow master hosts

The format of the shadow master host file is as follows:

• The first line of the file defines the primary master host
• The following lines define the shadow master hosts, one host per line

The order of the shadow master hosts is significant. The primary master host is the first line in the file. If the primary master host fails to proceed, the shadow master defined in the second line takes over. If this shadow master also fails, the shadow master defined in the third line takes over, and so forth.

Starting Shadow Master Hosts

To start a shadow sge_qmaster, the system must be sure either that the old sge_qmaster has terminated, or that it will terminate without performing actions that interfere with the newly started shadow sge_qmaster.

In very rare circumstances, you might not be able to determine that the old sge_qmaster has terminated or that it will terminate. In such cases, an error message is logged to the messages log file of the sge_shadowd daemons on the shadow master hosts. See Chapter 10, Fine Tuning, Error Messages, and Troubleshooting for further information.

Also, any attempts to open a tcp connection to a sge_qmaster daemon permanently fails. If this occurs, make sure that no master daemon is running, and then restart sge_qmaster manually on any of the shadow master machines. See Restarting Daemons From the Command Line for further details.

Configuring Shadow Master Hosts Environment Variables

Three environment variables affect the takeover time for a shadow master:

These variables interact in the following ways:

1. The master host updates the heartbeat file every 30 seconds.
2. The sge_shadowd daemon checks for changes to the heartbeat file every number of seconds defined by the SGE_CHECK_INTERVAL variable. This value must be greater than 30 seconds.
   • If the heartbeat file has been updated, the sge_shadowd daemon restarts the waiting clock.
   • If the heartbeat file has not been updated, the sge_shadowd daemon continues to wait until the number of seconds defined by the SGE_CHECK_INTERVAL variable expires. This action ensures that the sge_shadowd daemon is not too agressive in trying to take over and allows the master host some leeway in updating the heartbeat file.
3. When the SGE_GET_ACTIVE_INTERVAL has expired, the sge_shadowd daemon takes over if the heartbeat file is still not updated.

A reasonable configuration might be to set the SGE_CHECK_INTERVAL to 45 seconds and the SGE_GET_ACTIVE_INTERVAL to 90 seconds. So, after about 2 minutes, the takeover will occur. If you want to check the operation of the shadow host after you have configured these environment variables, you will have to disconnect the master host's network cable to simulate a failure.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Hosts With QMON

The QMON Host Configuration dialog box has four tabs:

• Execution Host tab
• Administration Host tab
• Submit Host tab
• Host Groups tab

The qconf command provides the command-line interface for managing host objects. See Configuring Hosts From the Command Line for more details.

Configuring Execution Hosts With QMON

Before you configure an execution host, you must first install the software on the execution host as described in How to Install Execution Hosts.

About the Execution Host Tab

To configure execution hosts, click the Host Configuration button on the QMON Main Control window, and then click the Execution Host tab. The Execution Host tab looks like the following figure:

Note Administrative or submit commands are allowed from execution hosts only if the execution hosts are also declared to be administration or submit hosts. See Configuring Administration Hosts With QMON and Configuring Submit Hosts With QMON.

Note the following in the Execution Host tab:

• The Hosts list displays the execution hosts that are already defined.
• The Load Scaling list displays the currently configured load-scaling factors for the selected execution host. See Load Parameters for information about load parameters.
• The Access Attributes list displays access permissions. See Managing User Access for information about access permissions.
• The Consumables/Fixed Attributes list displays resource availability for consumable and fixed resource attributes associated with the host. See Configuring Complex Resource Attributes for information about resource attributes.
• The Reporting Variables list displays the variables that are written to the reporting file when a load report is received from an execution host. See Defining Reporting Variables for information about reporting variables.
• The Usage Scaling list displays the current scaling factors for the individual usage metrics CPU, memory, and I/O for different machines. Resource usage is reported by sge_execd periodically for each currently running job. The scaling factors indicate the relative cost of resource usage on the particular machine for the user or project running a job. These factors could be used, for instance, to compare the cost of a second of CPU time on a 400 MHz processor to that of a 600 MHz CPU. Metrics that are not displayed in the Usage Scaling list have a scaling factor of 1.

How to Add or Modify an Execution Host

1. To add or modify an execution host, on the QMON Main Control window, click the Host Configuration button.
   
   
2. Click the Execution Host tab.
   
   
3. Click Add or Modify.
   The Add/Modify Exec Host dialog box appears as shown in the following figure. The Add/Modify Exec Host dialog box enables you to modify all attributes associated with an execution host. The name of an existing execution host is displayed in the Host field.
   
   
   
4. To add a new execution host, type its name in the Host field.
   
   
5. To define scaling factors, click the Scaling tab.
   • The Load column of the Load Scaling table lists all available load parameters. The Scale Factor column lists the corresponding definitions of the scaling. You can edit the Scale Factor column. Valid scaling factors are positive floating-point numbers in fixed-point notation or scientific notation.
   • The Usage column of the Usage Scaling table lists the current scaling factors for the usage metrics CPU, memory, and I/O. The Scale Factor column lists the corresponding definitions of the scaling. You can edit the Scale Factor column. Valid scaling factors are positive floating-point numbers in fixed-point notation or scientific notation.
     
6. To define the resource attributes to associate with the host, click the Consumables/Fixed Attributes tab.
   The Consumables/Fixed Attributes table lists all resource attributes for which a value is currently defined.
   
   

   Tip Use the Complex Configuration dialog box if you need more information about the current complex configuration, or if you want to modify it. For details about complex resource attributes, see Configuring Complex Resource Attributes.

   
   You can enhance the list by clicking either the Name or the Value column name. The Attribute Selection dialog box appears, which includes all resource attributes that are defined in the complex.
   

   • To add an attribute to the Consumables/Fixed Attributes table, select the attribute, and then click OK.
   • To modify an attribute value, double-click a Value field, and then type a value.
   • To delete an attribute, select the attribute, and then press Control-D or click mouse button 3.
     Click Ok to confirm that you want to delete the attribute.
     
7. Define access permissions based upon an existing project or upon user access lists.
   
   • To define user access permissions to the execution host based on previously configured user access lists, click the User Access tab.
     
     
   • To define project access permissions to the execution host based on previously configured projects, click the Project Access tab.
     
     
     
8. To define reporting variables, click the Reporting Variables tab.
   
   The Available list displays all the variables that can be written to the reporting file when a load report is received from the execution host.
   
   • To add a variable to the Selected list, select a reporting variable from the Available list, and then click the red right arrow .
     
   • To remove a reporting variable from the Selected list, select the variable, and then click the left red arrow.

How to Delete an Execution Host

1. To delete an execution host, on the QMON Main Control window, click the Host Configuration button.
   
   
2. Click the Execution Host tab.
   
   
3. In the Execution Host dialog box, select the host that you want to delete, and click Delete.

How to Shut Down an Execution Host Daemon

1. To shut down an execution host daemon, on the QMON Main Control window, click the Host Configuration button.
   
   
2. Click the Execution Host tab.
   
   
3. In the Execution Host dialog box, select a host, and click Shutdown.

Configuring Administration Hosts With QMON

Use the Administration Host tab to configure hosts on which administrative commands are allowed. The Host list displays the hosts that already have administrative permission.

How to Add or Remove an Administration Host With QMON

To configure Administration Hosts with QMON, do the following:

1. On the QMON Main Control window, click the Host Configuration button.
   The Host Configuration dialog box appears and displays the Administration Host tab as shown in the following figure.
   
   

   Note The Administration Host tab is displayed by default when you click the Host Configuration button for the first time.

   
   

2. To add a new administration host, type its name in the Host field, and then click Add, or press the Return key.
   
   
3. To delete an administration host, select the administrative host name from the list, and then click Delete.

Configuring Submit Hosts With QMON

Use the Submit Host tab to declare the hosts from which jobs can be submitted, monitored, and controlled. The Host list displays the hosts that already have submit permission.

Note No administrative commands are allowed from submit hosts unless the hosts are also declared to be administration hosts. See Configuring Administration Hosts With QMON for more information.

How to Add or Remove a Submit Host With QMON

1. On the QMON Main Control window, click the Host Configuration button.
   The Host Configuration dialog box appears.
   
   
2. Click the Submit Host tab.
   The Submit Host page appears as shown in the following figure.
   
   
   
3. To add a submit host, type the submit host name in the Host field, and then click Add, or press the Return key.
   
   
4. To delete a submit host, select the submit host name from the list, and then click Delete.

Configuring Host Groups With QMON

Use the Host Groups tab to configure host groups. The Hostgroup list displays the currently configured host groups. The Members list displays all the hosts that are members of the selected host group.

About the Host Groups Tab

To group similar hosts together, click the Host Configuration button on the QMON Main Control window, and then click the Host Groups tab.

Host groups enable you to use a single name to refer to multiple hosts. A host group can include other host groups as well as multiple individual hosts. Host groups that are members of another host group are subgroups of that host group.

For example, you might define a host group called @bigMachines that includes the following members:

• @solaris64
• @solaris32
• fangorn
• balrog

The initial @ sign indicates that the name is a host group. The host group @bigMachines includes all hosts that are members of the two subgroups @solaris64 and @solaris32. @bigMachines also includes two individual hosts, fangorn and balrog.

How to Add or Modify a Host Group With QMON

1. On the QMON Main Control window, click the Host Configuration button.
   The Host Configuration dialog box appears.
   
   
2. Click the Host Groups tab.
   The Host Groups page appears, as shown in the following figure.
   
   
   
3. To add a host group, click Add.
   The Add/Modify Host Group dialog box appears, as shown in the following figure.
   
   Type a host group name in the Hostgroup field.
   The host group name must begin with an @ sign.
   
   
4. To change information about a host group, select the host group in the Hostgroup list and click Modify.
   The Add/Modify Host Group dialog box appears and provides information about the selected host group.
   
   • To add a host to the Members list for the selected host group, type the host name in the Host field and click the red arrow.
   • To add a host group as a subgroup, select a host group name from the Defined Host Groups list and click the red arrow.
   • To remove a host or a host group, select it from the Members list and click the trash icon.
     
5. Click Ok to save your changes and close the dialog box.
   Click Cancel to close the dialog box without saving your changes.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Hosts From the Command Line

Configuring Execution Hosts From the Command Line

To configure execution hosts from the command line, use the following arguments for the qconf command:

• qconf -ae [ exec-host ]
  
  The -ae option (add execution host) displays an editor that contains a configuration template for an execution host. The editor is either the default vi editor or the editor that corresponds to the EDITOR environment variable. If you specify exec-host, which is the name of an already configured execution host, the configuration of this execution host is used as a template. To configure the execution host, change and save the template. See the host_conf(5) man page for a detailed description of the template entries to be changed.

• qconf -de hostname
  
  The -de option (delete execution host) deletes the specified host from the list of execution hosts. All entries in the execution host configuration are lost.

• qconf -me hostname
  
  The -me option (modify execution host) displays an editor that contains the configuration file template for the specified execution host. The editor is either the default vi editor or the editor that corresponds to the EDITOR environment variable. To modify the execution host configuration, change and save the configuration file template. See the host_conf(5) man page for a detailed description of the template entries to be changed.

• qconf -Me filename
  
  The -Me option (modify execution host) uses the content of filename as execution host configuration template. The configuration in the specified file must refer to an existing execution host. The configuration of this execution host is replaced by the file content. This qconf option is useful for changing the configuration of offline execution hosts, for example, in cron jobs, as the -Me option requires no manual interaction.

• qconf -se hostname
  
  The -se option (show execution host) shows the configuration of the specified execution host as defined in host_conf.

• qconf -sel
  
  The -sel option (show execution host list) displays a list of hosts that are configured as execution hosts.

Configuring Administration Hosts From the Command Line

To configure administration hosts from the command line, use the following arguments for the qconf command:

• qconf -ah hostname
  
  The -ah option (add administration host) adds the specified host to the list of administration hosts.

• qconf -dh hostname
  
  The -dh option (delete administration host) deletes the specified host from the list of administration hosts.

• qconf -sh
  
  The -sh option (show administration hosts) displays a list of all currently configured administration hosts.

Configuring Submit Hosts From the Command Line

To configure submit hosts from the command line, use the following arguments for the qconf command:

• qconf -as hostname
  
  The -as option (add submit host) adds the specified host to the list of submit hosts.

• qconf -ds hostname
  
  The -ds option (delete submit host) deletes the specified host from the list of submit hosts.

• qconf -ss
  
  The -ss option (show submit hosts) displays a list of the names of all currently configured submit hosts.

Configuring Host Groups From the Command Line

To configure host groups from the command line, use the following arguments for the qconf command:

• qconf -ahgrp host-group-name
  
  The -ahgrp option (add host group) adds a new host group to the list of host groups. See the hostgroup(5) man page for a detailed description of the configuration format.

• qconf -Ahgrp filename
  
  The -Ahgrp option (add host group from file) displays an editor that contains a host group configuration defined in filename. The editor is either the default vi editor or the editor that corresponds to the EDITOR environment variable. To configure the host group, change and save the configuration file template.

• qconf -dhgrp host-group-name
  
  The -dhgrp option (delete host group) deletes the specified host group from the list of host groups. All entries in the host group configuration are lost.

• qconf -mhgrp host-group-name
  
  The -mhgrp option (modify host group) displays an editor that contains the configuration of the specified host group as template. The editor is either the default vi editor or the editor that corresponds to the EDITOR environment variable. To modify the host group configuration, change and save the configuration file template.

• qconf -Mhgrp filename
  
  The -Mhgrp option (modify host group from file) uses the content of filename as host group configuration template. The configuration in the specified file must refer to an existing host group. The configuration of this host group is replaced by the file content.

• qconf -shgrp host-group-name
  
  The -shgrp option (show host group) shows the configuration of the specified host group.

• qconf -shgrp_tree host-group-name
  
  The -shgrp_tree option (show host group as tree) shows the configuration of the specified host group and its sub-hostgroups as a tree.

• qconf -shgrp_resolved host-group-name
  
  The -shgrp_resolved option (show host group with resolved host list) shows the configuration of the specified host group with a resolved host list.

• qconf -shgrpl
  
  The -shgrpl option (show host group list) displays a list of all host groups.

Monitoring Execution Hosts With qhost

Use the qhost command to retrieve a quick overview of the execution host status:

% qhost

This command produces output that is similar to the following example:

Example – Sample qhost Output

HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
-------------------------------------------------------------------------------
global                  -               -     -       -       -       -       -
arwen                   aix43           1     -       -       -       -       -
baumbart                irix65          2  0.00    1.1G   91.5M  128.0M     0.0
boromir                 hp11            1     -  128.0M       -  256.0M       -
carc                    lx24-amd64      2  0.00    3.8G  989.8M    1.0G     0.0
denethor                aix51           1 4.54G       -       -       -       -
durin                   lx24-x86        1  0.37  123.1M   46.5M  213.6M   26.6M
eomer                   sol-sparc64     1  0.13  256.0M  248.0M  513.0M   93.0M
lolek                   tru64           1  0.02    1.0G  790.0M    1.0G    8.0K
mungo                   lx22-alpha      1  1.00  248.9M   78.8M  129.8M    2.5M
nori                    sol-x86         2  0.38 1023.0M  372.0M  512.0M   37.0M
pippin                  darwin          1  0.00  640.0M  264.0M     0.0     0.0
smeagol                 hp11            1  0.35  512.0M  425.0M    1.0G   95.0M

See the qhost(1) man page for a description of the output format and for more options.

Killing Daemons From the Command Line

To kill Grid Engine system daemons from the command line, use one of the following commands:

% qconf -ke[j] {<hostname>[,...] | all}
% qconf -ks
% qconf -km

You must have manager or operator privileges to use these commands. See Managing Users Access for more information about manager and operator privileges.

• The qconf -ke command shuts down the execution daemons. However, it does not cancel active jobs. Jobs that finish while no sge_execd is running on a system are not reported to sge_qmaster until sge_execd is restarted. The job reports are not lost, however.
  The qconf -kej command kills all currently active jobs and brings down all execution daemons.
  Use a comma-separated list of the execution hosts you want to shut down, or specify all to shut down all execution hosts in the cluster.
• The qconf -ks command shuts down the scheduler sge_schedd.
• The qconf -km command forces the sge_qmaster process to terminate.

If you want to wait for any active jobs to finish before you run the shutdown procedure, use the qmod -dq command for each cluster queue, queue instance, or queue domain before you run the qconf sequence described above. For information about cluster queues, queue instances, and queue domains, see About Configuring Queues.

% qmod -dq {<cluster-queue> | <queue-instance> | <queue-domain>}

The qmod -dq command prevents new jobs from being scheduled to the disabled queue instances. You should then wait until no jobs are running in the queue instances before you kill the daemons.

Restarting Daemons From the Command Line

Log in as root on the machine on which you want to restart Grid Engine system daemons.
Type the following commands to run the startup scripts:

% $SGE_ROOT/$SGE_CELL/common/sgemaster
% $SGE_ROOT/$SGE_CELL/common/sgeexecd

These scripts looks for the daemons that are normally running on this host and then starts them.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Queues

This section provides the following information about configuring queues:

For information about configuring queue calendars, see Configuring Queue Calendars.

About Configuring Queues

Queues are containers for different categories of jobs. Queues provide the corresponding resources for concurrent execution of multiple jobs that belong to the same category.

In Sun Grid Engine, you can associate a queue with one host or with multiple hosts. Because queues can extend across multiple hosts, they are called cluster queues. Cluster queues enable you to manage a cluster of execution hosts by means of a single cluster queue configuration.

Each host that is associated with a cluster queue receives an instance of that cluster queue, which resides on that host. This guide refers to these instances as queue instances. Within any cluster queue, you can configure each queue instance separately. By configuring individual queue instances, you can manage a heterogeneous cluster of execution hosts by means of a single cluster queue configuration.

When you modify a cluster queue, all of its queue instances are modified simultaneously. Within a single cluster queue, you can specify differences in the configuration of queue instances. Consequently, a typical setup might have only a few cluster queues, and the queue instances controlled by those cluster queues remain largely in the background.

Note The distinction between cluster queues and queue instances is important. For example, jobs always run in queue instances, not in cluster queues.

When you configure a cluster queue, you can associate any combination of the following host objects with the cluster queue:

• One execution host
• A list of separate execution hosts
• One or more host groups

Note To enable a queue to operate correctly in a parallel environment (PE), you must associate the queue with the PE. This association enables more control of the resources and lets you assign specific queues to handle the parallel workload.

Use the queue_conf(5) attribute pe_list to identify the suited PEs. Then, to link the PE and queues, use either the QMON utility or the following form of the qconf command:

# qconf -mq <queue_name>

A host group is a group of hosts that can be treated collectively as identical. Host groups enable you to manage multiple hosts by means of a single host group configuration. For more information about host groups, see Configuring Host Groups With QMON.

When you associate individual hosts with a cluster queue, the name of the resulting queue instance on each host combines the cluster queue name with the host name. The cluster queue name and the host name are separated by an @ sign. For example, if you associate the host myexechost with the cluster queue myqueue, the name of the queue instance on myexechost is myqueue@myexechost.

When you associate a host group with a cluster queue, you create what is known as a queue domain. Queue domains enable you to manage groups of queue instances that are part of the same cluster queue and whose assigned hosts are part of the same host group. A queue domain name combines a cluster queue name with a host group name, separated by an @ sign. For example, if you associate the host group myhostgroup with the cluster queue myqueue, the name of the queue domain is myqueue@@myhostgroup.

Note Queue domain names always include two @ signs, because all host group names begin with an @ sign.

Jobs do not wait in queue instances. Jobs start running immediately as soon as they are dispatched. The scheduler's list of pending jobs is the only waiting area for jobs.

Configuring queues registers the queue attributes with sge_qmaster. As soon as queues are configured, they are instantly visibly to the whole cluster and to all users on all hosts belonging to the Grid Engine system.

For further details, see the queue_conf(5) man page.

How to Configure Queues With QMON

1. On the QMON Main Control window, click the Queue Control button.
   The Cluster Queues dialog box appears, as shown in the following example:
   
   The Cluster Queues dialog box and its facilities for monitoring and manipulating the status of cluster queues and queue instances are described in Monitoring and Controlling Queues With QMON.
   
   
2. Choose from the following actions:
   • To add a new cluster queue, click Add.
     When you click Add, the Queue Configuration – Add dialog box appears. If you are adding a new cluster queue, you must specify a queue name and the names of the hosts on which the queue instances are to reside.
   • To modify an existing cluster queue, select it from the Cluster Queue list, and then click Modify.
     When you click Modify, the Modify queue-name dialog box appears. If you are modifying an existing queue, the name of the queue is displayed in the Queue Name field. The hosts where the queue instances reside are displayed in the Hostlist field.
   • To import all parameters of an existing cluster queue, click Clone.
     You select the queue you want to clone from a list of existing queues.
     Whether you add, modify, or clone a cluster queue, the resulting Queue Configuration dialog box initially displays the General Configuration tab.
     In the Hostlist field, you can specify the names of individual hosts. You can also specify the names of previously defined host groups. Queue instances of this cluster queue will reside on all individual hosts and on all members of the host groups that you specify, including all members of any host subgroups. For more information about host groups, see Configuring Host Groups With QMON.
     The following 11 tabs for specifying parameter sets are available to define a queue:
     • General Configuration – see How to Configure General Parameters
     • Execution Method – see How to Configure Execution Method Parameters
     • Checkpointing – see How to Configure the Checkpointing Parameters
     • Parallel Environment – see How to Configure Parallel Environments
     • Load/Suspend Thresholds – see How to Configure Load and Suspend Thresholds
     • Limits – see How to Configure Limits
     • Complex – see How to Configure Complex Resource Attributes
     • Subordinates – see How to Configure Subordinate Queues
     • User Access – see How to Configure User Access Parameters
     • Project Access – see How to Configure Project Access Parameters
     • Owners – see How to Configure Owners Parameters
       
       
3. To set default parameters for the cluster queue, select @/ in the Attributes for Host/Hostgroup list, and then click the tab containing the parameters that you want to set.
   Default parameters are set for all queue instances on all hosts listed under Hostlist. You can override the default parameter values on a host or a host group that you specify. To set override parameters for a host or a host group, first select the name from the Attributes for Host/Hostgroup list. Then click the tab containing the parameters that you want to set. The values of the parameters that you set override the cluster queue's default parameters on the selected host or host group.
   
   
4. To set a host-specific parameter, you must first enable the parameter for configuration.
   Click the lock icon at the left of the parameter that you want to set, and then change the parameter's value.
   
   
5. To load the settings of other objects that were modified while the Queue Configuration dialog box was open, click the Refresh button.
   
   
6. Click OK to register all queue configuration changes with sge_qmaster and close the dialog box.
   Click Cancel to close the dialog box without saving your changes.

How to Configure General Parameters

1. To configure general parameters, click the General Configuration tab.
   The General Configuration tab is shown in the figure above.
   
   
2. Specify one or more of the following parameters:
   • Sequence Nr – The sequence number of the queue.
   • Processors – A specifier for the processor set to be used by the jobs running in that queue. For some operating system architectures, this specifier can be a range, such as 1-4,8,10, or just an integer identifier of the processor set. See the arc_depend_*.asc files in the doc directory of your Grid Engine software distribution for more information.

     Caution Do not change this value unless you are certain that you need to change it.

   • tmp Directory – Temporary directory path.
   • Shell – Default command interpreter to use for running the job scripts.
   • Shell Start Mode – The mode in which to start the job script.
   • Initial State – The state in which a newly added queue comes up. Also, the state in which a queue instance is restored if the sge_execd running on the queue instance host gets restarted.
   • Rerun Jobs – The queue's default rerun policy to be enforced on jobs that were aborted, for example, due to system crashes. The user can override this policy using the qsub -r command or the Submit Job dialog box. See the Extended Job Example.
   • Calendar – A calendar attached to the queue. This calendar defines on-duty and off-duty times for the queue.
   • Notify Time – The time to wait between delivery of SIGUSR1/SIGUSR2 notification signals and suspend or kill signals.
   • Job's Nice – The nice value with which to start the jobs in this queue. 0 means use the system default.
   • Slots – The number of jobs that are allowed to run concurrently in the queue. Slots are also referred to as job slots.
   • Type – The type of the queue and of the jobs that are allowed to run in this queue. Type can be Batch, Interactive, or both.
     See the queue_conf(5) man page for detailed information about these parameters.

How to Configure Execution Method Parameters

1. To configure execution method parameters, click the Execution Method tab.
   The Execution Method tab is shown in the following figure.
   
   
   
2. Specify one or more of the following parameters:
   • Prolog – A queue-specific prolog script. The prolog script is run with the same environment as the job before the job script is started.
   • Epilog – A queue-specific epilog script. The epilog script is run with the same environment as the job after the job is finished.
   • Starter Method, Suspend Method, Resume Method, Terminate Method – Use these fields to override the default methods for applying these actions to jobs.
     See the queue_conf(5) man page for detailed information about these parameters.

How to Configure the Checkpointing Parameters

1. To configure the checkpointing parameters, click the Checkpointing tab.
   The Checkpointing tab is shown in the following figure.
   
   
   
2. Specify one or more of the following parameters:
   • MinCpuTime – The periodic checkpoint interval.
   • Referenced Ckpt Objects – A list of checkpointing environments associated with the queue.
     
     
3. To reference a checkpointing environment from the queue, select the name of a checkpointing environment from the Available list, and then click the right arrow to add it to the Referenced list.
   
   
4. To remove a checkpointing environment from the Referenced list, select it, and then click the left arrow.
   
   
5. To add or modify checkpointing environments, click the button below the red arrows to open the Checkpointing Configuration dialog box.
   For more information, see Configuring Checkpointing Environments With QMON.
   See the queue_conf(5) man page for detailed information about these parameters.

How to Configure Parallel Environments

1. To configure parallel environments, click the Parallel Environment tab.
   The Parallel Environment tab is shown in the following figure.
   
   
   
2. To reference a parallel environment from the queue, select the name of a parallel environment from the Available PEs list, and then click the right arrow to add it to the Referenced PEs list.
   
   
3. To remove a checkpointing environment from the Referenced PEs list, select it, and then click the left arrow.
   
   
4. To add or modify parallel environments, click the button below the red arrows to open the Parallel Environment Configuration dialog box.
   For more information, see Configuring Parallel Environments With QMON.
   See the queue_conf(5) man page for detailed information about this parameter.

How to Configure Load and Suspend Thresholds

1. To configure load and suspend thresholds, click the Load/Suspend Thresholds tab.
   The Load/Suspend Thresholds tab is shown in the following figure.
   
   
   • The Load Thresholds and the Suspend Thresholds tables display the defined overload thresholds for load parameters and consumable resource attributes. See Configuring Complex Resource Attributes for more information.
     In the case of load thresholds, overload prevents the queue from receiving further jobs. In the case of suspend thresholds, overload suspends jobs in the queue to reduce the load.
     
   • Suspend interval. The time interval between suspension of other jobs in case the suspend thresholds are still exceeded.
   • Jobs suspended per interval. The number of jobs to suspend per time interval in order to reduce the load on the system that is hosting the configured queue.
     See the queue_conf(5) man page for detailed information about these parameters.
     
     
2. To change an existing threshold, select it, and then double-click the corresponding Value field.
   
   
3. To add new thresholds, click Load or Value.
   A selection list appears with all valid attributes that are attached to the queue.
   To add an attribute to the Load column of the corresponding threshold table, select an attribute, and then click OK.
   
   
4. To delete an existing threshold, select it, and then type Ctrl-D or click mouse button 3.
   You are prompted to confirm that you want to delete the selection.

How to Configure Limits

1. To configure hard or soft limit parameters, click the Limits tab.
   When a hard limit is exceeded, the running job in the queue is stopped immediately. When a soft limit is exceeded, a signal is sent that the job can intercept before the job is stopped. The Limits tab is shown in the following figure.
   
   
   
2. To change a value of a limit, click the button at the right of the field whose value you want to change.
   A dialog box appears where you can type either Memory or Time limit values.
   See the queue_conf(5) and the setrlimit(2) man pages for detailed information about limit parameters and their interpretation for different operating system architectures.

How to Configure Complex Resource Attributes

1. To configure resource attributes, click the Complex tab.
   The Complex tab is shown in the following figure.
   
   The attributes for which values are explicitly defined are displayed in the Consumable/Fixed Attributes table. The available resource attributes are assembled by default from the complex.
   Resource attributes are either consumable or fixed. The definition of a consumable value defines a capacity managed by the queue. The definition of a fixed value defines a queue-specific value. See Configuring Complex Resource Attributes for further details.
   
   
2. To change an attribute, select it, and then double-click the corresponding Value field.
   
   
3. To add new attribute definitions, click Load or Value.
   The Attribute Selection dialog box appears with a list of all valid attributes that are attached to the queue.
   • To add an attribute to the Load column of the attribute table, select it, and then click OK.
   • To delete an attribute, select it, and then press Ctrl-D or click mouse button 3. You are prompted to confirm that you want to delete the attribute.
     See the queue_conf(5) page for detailed information about these attributes.

Next Steps

Use the Complex Configuration dialog box to check or modify the current complex configuration before you attach user-defined resource attributes to a queue or before you detach them from a queue. To access the Complex Configuration dialog box, click the Complex Configuration button on the QMON Main Control window. For more information, see See Configuring Complex Resource Attributes.

How to Configure Subordinate Queues

1. To configure subordinate queues, click the Subordinates tab.
   Use the subordinate queue facility to implement high priority and low priority queues as well as standalone queues. The Subordinates tab is shown in the following figure.
   
   
   
2. Specify the following parameters:
   • Queue – A list of the queues that are subordinated to the configured queue.
     Subordinated queue instances on a host are suspended if the configured queue instance on the same host becomes busy. Subordinated queue instances on that host are resumed when the configured queue instance is no longer busy.
   • Max Slots – For any subordinated queue, you can configure the number of job slots that must be filled in the configured queue to trigger a suspension. If no maximum slot value is specified, all job slots must be filled to trigger suspension of the corresponding queue.
     See the queue_conf(5) man page for detailed information about these parameters.

How to Configure User Access Parameters

1. To configure user access parameters, click the User Access tab.
   The User Access tab is shown in the following figure.
   
   Users or user groups belonging to access lists that are included in the Allow Access list have access to the queue. Users who are included in the Deny Access list cannot access the queue. If the Allow Access list is empty, access is unrestricted unless explicitly stated otherwise in the Deny Access list.
   
   
2. To add or modify user access lists, click the button between the Available Access Lists and the Allow Access and Deny Access lists to open the User Configuration dialog box.
   For more information about the User Configuration dialog box, see How to Configure User Access Lists With QMON.
   See the queue_conf(5) man page for detailed information about these parameters.

How to Configure Project Access Parameters

1. To configure project access parameters, click the Project Access tab.
   The Project Access tab is shown in the following figure.
   
   Jobs that are submitted to a project that belongs to the list of allowed projects have access to the queue. Jobs that are submitted to denied projects are not dispatched to the queue.
   
   
2. To add or modify project access, click the button between the Available Projects list and the Allow Project Access and Deny Project Access lists to open the Project Configuration dialog box.
   For more information, see How to Define Projects With QMON.
   See the queue_conf(5) man page for detailed information about these parameters.

How to Configure Owners Parameters

1. To configure owners parameters, click the Owners tab.
   The Owners tab is shown in the following figure.
   
   
   
2. Select queue owners in the Owner List.
   You can add any user account to the owner list. Typically, you assign users to be owners of certain queue instances to enable them to suspend or disable jobs when necessary. For example, users might occasionally need certain machines for important work, and those machines might be strongly affected by jobs that are running in the background.
   Queue owners can do the following:
   • Suspend – Stop execution of all jobs running in the queue and close the queue.

     Note Jobs that are suspended explicitly while a queue is suspended are not resumed when the queue is resumed. Explicitly suspended jobs must be resumed explicitly.

   • Resume – Unsuspend the queue, and then open it.
   • Disable – Close the queue, but do not affect running jobs.
   • Enable – Open the queue.
     See the queue_conf(5) man page for detailed information about these parameters.
     
     
3. To delete a user account from the queue owner list, select it, and then click the trash can icon.

Configuring Queues From the Command Line

To configure queues from the command line, type the following command with the appropriate options:

# qconf <options>

The qconf command has the following options:

• qconf -aq [cluster-queue]
  The -aq option (add cluster queue) displays an editor containing a template for cluster queue configuration. The editor is either the default vi editor or an editor defined by the EDITOR environment variable. If cluster-queue is specified, the configuration of this cluster queue is used as template. Configure the cluster queue by changing the template and then saving it. See the queue_conf(5) man page for a detailed description of the template entries to change.
• qconf -Aq filename
  The -Aq option (add cluster queue from file) uses the file filename to define a cluster queue. The definition file might have been produced by the qconf -sq queue command.
• qconf -cq queue [,...]
  The -cq option (clean queue) cleans the status of the specified cluster queues, queue domains, or queue instances to be idle and free from running jobs. The status is reset without respect to the current status. This option is useful for eliminating error conditions, but you should not use it in normal operation mode.
• qconf -dq cluster-queue [,...]
  The -dq option (delete cluster queue) deletes the cluster queues specified in the argument list from the list of available queues.
• qconf -mq cluster-queue
  The -mq option (modify cluster queue) modifies the specified cluster queue. The -mq option displays an editor containing the configuration of the cluster queue to be changed. The editor is either the default vi editor or an editor defined by the EDITOR environment variable. Modify the cluster queue by changing the configuration and then saving your changes.
• qconf -Mq filename
  The -Mq option (modify cluster queue from file) uses the file filename to define the modified cluster queue configuration. The definition file might have been produced by the qconf -sq queue command and subsequent modification.
• qconf -sq [queue[,...]]
  The -sq option (show queue) without arguments displays the default template cluster queue, queue domain, or queue instance configuration. The -sq option with arguments displays the current configuration of the specified queues.
• qconf -sql
  The -sql option (show cluster queue list) displays a list of all currently configured cluster queues.

The qconf command provides the following set of options that you can use to change specific queue attributes:

• -aattr – Add attributes
• -Aattr – Add attributes from a file
• -dattr – Delete attributes
• -Dattr – Delete attributes listed in a file
• -mattr – Modify attributes
• -Mattr – Modify attributes from a file
• -rattr – Replace attributes
• -Rattr – Replace attributes from a file
• -sobjl – Show list of configuration objects

For a description of how to use these options and for some examples of their use, see Using Files to Modify Queues, Hosts, and Environments. For detailed information about these options, see the qconf(1) man page.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Queue Calendars

For information about configuring queues, see Configuring Queues.

About Queue Calendars

Queue calendars define the availability of queues according to the day of the year, the day of the week, or the time of day. You can configure queues to change their status at specified times. You can change the queue status to disabled, enabled, suspended, or resumed (unsuspended).

The Grid Engine system enables you to define a site-specific set of calendars, each of which specifies status changes and the times at which the changes occur. These calendars can be associated with queues. Each queue can attach a single calendar, thereby adopting the availability profile defined in the attached calendar.

The syntax of the calendar format is described in detail in the calendar_conf(5) man page. A few examples are given in the next sections, along with a description of the corresponding administration facilities.

How to Configure Queue Calendars With QMON

1. In the QMON Main Control window, click the Calendar Configuration button.
   The Calendar Configuration dialog box appears. The Calendars list displays the available calendars.
   
   
   
2. In the Calendars list, click the calendar configuration that you want to modify or delete.
   • To delete the selected calendar, click Delete.
   • To modify the selected calendar, click Modify.
   • To add a calendar, click Add.
     In all cases, the Add/Modify Calendar dialog box appears. The following example shows the definition for a queue that is available outside office hours and on weekends. In addition, the Christmas holidays are defined to be handled like weekends.
     
     If you click Modify or Delete, the Calendar Name field displays the name of the selected calendar. If you click Add, you need to add a name in the Calendar Name field.
     The Year and Week fields enable you to define the calendar events, using the syntax described in the calendar_conf(5) man page.
     See the calendar_conf(5) man page for a detailed description of the syntax and for more examples.
     
     
3. To control access to a specific queue based on calendar data, attach the calendar to the queue as follows:
   1. From the main QMON window, click Queue Control.
   2. Select the queue name to which you want to attach the calendar and click the Modify button. The Modify queuename window appears with the General Configuration tab selected.
   3. On the General Configuration tab, use the drop-down list next to the Calendar field to choose the calendar to attach. Attaching a calendar configuration to a queue sets the availability profile defined by the calendar for the queue. See About Configuring Queues for more details about configuring queues.
      

Configuring Queue Calendars From the Command Line

To configure queue calendars from the command line, type the following command with appropriate options:

% qconf <options>

The following options are available:

• qconf -acal calendar-name – The -acal option (add calendar) adds a new calendar configuration named calendar-name to the cluster. An editor with a template configuration appears, enabling you to define the calendar.
• qconf -Acal filename – The -Acal option (add calendar from file) adds a new calendar configuration to the cluster. The added calendar is read from the specified file.
• qconf -dcal calendar-name [,...] – The -dcal option (delete calendar) deletes the specified calendar.
• qconf -mcal calendar-name – The -mcal option (modify calendar) modifies an existing calendar configuration named calendar-name. An editor opens calendar-name, enabling you to make changes to the definition.
• qconf -Mcal filename – The -Mcal option (modify calendar from file) modifies an existing calendar configuration. The calendar to modify is read from the specified file.
• qconf -scal calendar-name – The -scal option (show calendar) displays the configuration for calendar-name.
• qconf -scall – The -scall option (show calendar list) displays a list of all configured calendars.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Managing the Scheduler

This section contains information about scheduling Grid Engine system policy implementation through the scheduler. The following topics are covered:

For information about Grid Engine policies, see Managing Policies.

Administering the Scheduler

This section describes how the Grid Engine system schedules jobs for execution, describes different types of scheduling strategies, and explains how to configure the scheduler.

About Scheduling

The Grid Engine system includes the following job-scheduling activities:

• Predispatching decisions – Activities such as eliminating queues because they are full or overloaded, spooling jobs that are currently not eligible for execution, and reserving resources for higher-priority jobs
• Dispatching – Deciding a job's importance with respect to other pending jobs and running jobs, sensing the load on all machines in the cluster, and sending the job to a queue on a machine selected according to configured selection criteria
• Postdispatch monitoring – Adjusting a job's relative importance as it gets resources and as other jobs with their own relative importance enter or leave the system

The Grid Engine software schedules jobs across a heterogeneous cluster of computers, based on the following criteria:

• The cluster's current load
• The jobs' relative importance
• The hosts' relative performance
• The jobs' resource requirements, for example, CPU, memory, and I/O bandwidth

Decisions about scheduling are based on the strategy for the site and on the instantaneous load characteristics of each computer in the cluster. A site's scheduling strategy is expressed through the Grid Engine system's configuration parameters. Load characteristics are ascertained by collecting performance data as the system runs.

Scheduling Strategies

The administrator can set up strategies with respect to the following scheduling tasks:

• Dynamic resource management – The Grid Engine system dynamically controls and adjusts the resource entitlements that are allocated to running jobs. In other words, the system modifies their CPU share.
• Queue sorting – The software ranks the queues in the cluster according to the order in which the queues should be filled up.
• Job sorting – Job sorting determines the order in which the Grid Engine system attempts to schedule jobs.
• Resource reservation and backfilling – Resource reservation reserves resources for jobs, blocking their use by jobs of lower priority. Backfilling enables lower-priority jobs to use blocked resources when using those resources does not interfere with the reservation.

Dynamic Resource Management

The Grid Engine software uses a weighted combination of the following three ticket-based policies to implement automated job scheduling strategies:

• Share-based
• Functional (sometimes called Priority)
• Override

You can set up the Grid Engine system to routinely use either a share-based policy, a functional policy, or both. You can combine these policies in any combination. For example, you could give zero weight to one policy and use only the second policy. Or you could give both policies equal weight.

Along with routine policies, administrators can also override share-based and functional scheduling temporarily or, for certain purposes such as express queues, permanently. You can apply an override to one job or to all jobs associated with a user, a department, a project, or a job class (that is, a queue).

In addition to the three policies for mediating among all jobs, the Grid Engine system sometimes lets users set priorities among the jobs they own. For example, a user might say that jobs one and two are equally important, but that job three is more important than either job one or job two. Users can set their own job priorities if the combination of policies includes the share-based policy, the functional policy, or both. Also, functional tickets must be granted to jobs.

Tickets

The share-based, functional, and override scheduling policies are implemented with tickets. Each policy has a pool of tickets. A policy allocates tickets to jobs as the jobs enter the multimachine Grid Engine system. Each routine policy that is in force allocates some tickets to each new job. The policy might also reallocate tickets to running jobs at each scheduling interval.

Tickets weight the three policies. For example, if no tickets are allocated to the functional policy, that policy is not used. If the functional ticket pool and the share-based ticket pool have an equal number of tickets, both policies have equal weight in determining a job's importance.

Tickets are allocated to the routine policies at system configuration by Grid Engine system managers. Managers and operators can change ticket allocations at any time with immediate effect. Additional tickets are injected into the system temporarily to indicate an override. Policies are combined by assignment of tickets. When tickets are allocated to multiple policies, a job gets a portion of each policy's tickets, which indicates the job's importance in each policy in force.

The Grid Engine system grants tickets to jobs that are entering the system to indicate their importance under each policy in force. At each scheduling interval, each running job can gain tickets, lose tickets, or keep the same number of tickets. For example, a job might gain tickets from an override. A job might lose tickets because it is getting more than its fair share of resources. The number of tickets that a job holds represents the resource share that the Grid Engine system tries to grant that job during each scheduling interval.

You configure a site's dynamic resource management strategy during installation. First, you allocate tickets to the share-based policy and to the functional policy. You then define the share tree and the functional shares. The share-based ticket allocation and the functional ticket allocation can change automatically at any time. The administrator manually assigns or removes tickets.

Queue Sorting

The following means are provided to determine the order in which the Grid Engine system attempts to fill up queues:

• Load reporting – Administrators can select which load parameters are used to compare the load status of hosts and their queue instances. The wide range of standard load parameters that are available, and an interface for extending this set with site-specific load sensors, are described in Configuring Complex Resource Attributes With QMON.
• Load scaling – Load reports from different hosts can be normalized to reflect a comparable situation. See Configuring Execution Hosts With QMON for further details.
• Load adjustment – The Grid Engine software can be configured to automatically correct the last reported load as jobs are dispatched to hosts. The corrected load represents the expected increase in the load situation caused by recently started jobs. This artificial increase of load can be automatically reduced as the load impact of these jobs takes effect.
• Sequence number – Queues can be sorted following a strict sequence.

Job Sorting

Before the Grid Engine system starts to dispatch jobs, the jobs are brought into priority order, highest priority first. The system then attempts to find suitable resources for the jobs in priority sequence.

Without any administrator influence, the order is first-in, first-out (FIFO). The administrator has the following means to control the job order:

• Ticket-based job priority – Jobs are always treated according to their relative importance as defined by the number of tickets that the jobs have. Pending jobs are sorted in ticket order. Any change that the administrator applies to the ticket policy also changes the sorting order.
• Urgency-based job priority – Jobs can have an urgency value that determines their relative importance. Pending jobs are sorted according to their urgency value. Any change applied to the urgency policy also changes the sorting order.
• POSIX priority – You can use the -p option to the qsub command to implement site-specific priority policies. The -p option specifies a range of priorities from -1023 to 1024. The higher the number, the higher the priority. The default priority for jobs is zero.
• Maximum number of user or user group jobs – You can restrict the maximum number of jobs that a user or a UNIX user group can run concurrently. This restriction influences the sorting order of the pending job list, because the jobs of users who have not exceeded their limit are given preference.

For each priority type, a weighting factor can be specified. This weighting factor determines the degree to which each type of priority affects overall job priority. To make it easier to control the range of values for each priority type, normalized values are used instead of the raw ticket values, urgency values, and POSIX priority values.

The following formula expresses how a job's priority values are determined:

job_priority = weight_urgency * normalized_urgency_value + 
weight_ticket * normalized_ticket_value + 
weight_priority * normalized_POSIX_priority_value

You can use the qstat command to monitor job priorities:

• Use qstat -pri to monitor job priorities overall, including POSIX priority.
• Use qstat -ext to monitor job priorities based on the ticket policy.
• Use qstat -urg to monitor job priorities based on the urgency policy.
• Use qstat -pri to diagnose job priority issues when urgency policy, ticket based policies and -p <priority> are used concurrently
• Use qstat -explain to diagnose various queue instance based error conditions.

About the Urgency Policy

The urgency policy defines an urgency value for each job. The urgency value is derived from the sum of three contributions:

• Resource requirement contribution
• Waiting time contribution
• Deadline contribution

The resource requirement contribution is derived from the sum of all hard resource requests, one addend for each request.

If the resource request is of the type numeric, the resource request addend is the product of the following three elements:

• The resource's urgency value as defined in the complex. For more information, see Configuring Complex Resource Attributes With QMON.
• The assumed slot allocation of the job.
• The per slot request specified by the qsub -l command.

If the resource request is of the type string, the resource request addend is the resource's urgency value as defined in the complex.

The waiting time contribution is the product of the job's waiting time, in seconds, and the waiting-weight value specified in the Policy Configuration dialog box.

The deadline contribution is zero for jobs without a deadline. For jobs with a deadline, the deadline contribution is the weight-deadline value, which is defined in the Policy Configuration dialog box, divided by the free time, in seconds, until the deadline initiation time.

For information about configuring the urgency policy, see Configuring the Urgency Policy.

Resource Reservation and Backfilling

Resource reservation enables you to reserve system resources for specified pending jobs. When you reserve resources for a job, those resources are blocked from being used by jobs with lower priority.

Jobs can reserve resources depending on criteria such as resource requirements, job priority, waiting time, resource sharing entitlements, and so forth. The scheduler enforces reservations in such a way that jobs with the highest priority get the earliest possible resource assignment. This avoids such well-known problems as "job starvation."

You can use resource reservation to guarantee that resources are dedicated to jobs in job-priority order.

Consider the following example. Job A is a large pending job, possibly parallel, that requires a large amount of a particular resource. A stream of smaller Jobs B(i) require a smaller amount of the same resource. Without resource reservation, a resource assignment for Job A cannot be guaranteed, assuming that the stream of B(i) jobs does not stop. The resource cannot be guaranteed even though Job A has a higher priority than the B(i) jobs.

With resource reservation, Job A gets a reservation that blocks the lower priority Jobs B(i). Resources are guaranteed to be available for Job A as soon as possible.

Backfilling enables a lower-priority job to use resources that are blocked due to a resource reservation. Backfilling work only if there is a runnable job whose prospective run time is small enough to allow the blocked resource to be used without interfering with the original reservation.

In the example described earlier, a Job C, of very short duration, could use backfilling to start before Job A.

Because resource reservation causes the scheduler to look ahead, using resource reservation affects system performance. In a small cluster, the effect on performance is negligible when there are only a few pending jobs. In larger clusters, however, and in clusters with many pending jobs, the effect on performance might be significant.

To offset this potential performance degradation, you can limit the overall number of resource reservations that can be made during a scheduling interval. You can limit resource reservation in two ways:

• To limit the absolute number of reservations that can be made during a scheduling interval, set the Maximum Reservation parameter on the Scheduler Configuration dialog box. For example, if you set Maximum Reservation to 20, no more than 20 reservations can be made within an interval.
• To limit reservation scheduling to only those jobs that are important, use the -R y option of the qsub command. In the example described earlier, there is no need to schedule B(i) job reservations just for the sake of guaranteeing the resource reservation for Job A. Job A is the only job that you need to submit with the -R y option.

You can configure the scheduler to monitor how it is influenced by resource reservation. When you monitor the scheduler, information about each scheduling run is recorded in the file $SGE_ROOT/$SGE_CELL/common/schedule.

The following example shows what schedule monitoring does. Assume that the following sequence of jobs is submitted to a cluster where the global license consumable resource is limited to 5 licenses:

qsub -N L4_RR -R y -l h_rt=30,license=4 -p 100 $SGE_ROOT/examples/jobs/sleeper.sh 20
qsub -N L5_RR -R y -l h_rt-30,license=5        $SGE_ROOT/examples/jobs/sleeper.sh 20
qsub -N L1_RR -R y -l h_rt=31,license=1        $SGE_ROOT/examples/jobs/sleeper.sh 20

Assume that the default priority settings in the scheduler configuration are being used:

weight_priority          1.000000
weight_urgency           0.100000
weight_ticket            0.010000

The -p 100 priority of job L4_RR supersedes the license-based urgency, which results in the following prioritization:

job-ID  prior    name
---------------------
   3127 1.08000 L4_RR
   3128 0.10500 L5_RR
   3129 0.00500 L1_RR

In this case, traces of these jobs can be found in the schedule file for six schedule intervals:

::::::::
      3127:1:STARTING:1077903416:30:G:global:license:4.000000
      3127:1:STARTING:1077903416:30:Q:all.q@carc:slots:1.000000
      3128:1:RESERVING:1077903446:30:G:global:license:5.000000
      3128:1:RESERVING:1077903446:30:Q:all.q@bilbur:slots:1.000000
      3129:1:RESERVING:1077903476:31:G:global:license:1.000000
      3129:1:RESERVING:1077903476:31:Q:all.q@es-ergb01-01:slots:1.000000
      ::::::::
      3127:1:RUNNING:1077903416:30:G:global:license:4.000000
      3127:1:RUNNING:1077903416:30:Q:all.q@carc:slots:1.000000
      3128:1:RESERVING:1077903446:30:G:global:license:5.000000
      3128:1:RESERVING:1077903446:30:Q:all.q@es-ergb01-01:slots:1.000000
      3129:1:RESERVING:1077903476:31:G:global:license:1.000000
      3129:1:RESERVING:1077903476:31:Q:all.q@es-ergb01-01:slots:1.000000
      ::::::::
      3128:1:STARTING:1077903448:30:G:global:license:5.000000
      3128:1:STARTING:1077903448:30:Q:all.q@carc:slots:1.000000
      3129:1:RESERVING:1077903478:31:G:global:license:1.000000
      3129:1:RESERVING:1077903478:31:Q:all.q@bilbur:slots:1.000000
      ::::::::
      3128:1:RUNNING:1077903448:30:G:global:license:5.000000
      3128:1:RUNNING:1077903448:30:Q:all.q@carc:slots:1.000000
      3129:1:RESERVING:1077903478:31:G:global:license:1.000000
      3129:1:RESERVING:1077903478:31:Q:all.q@es-ergb01-01:slots:1.000000
      ::::::::
      3129:1:STARTING:1077903480:31:G:global:license:1.000000
      3129:1:STARTING:1077903480:31:Q:all.q@carc:slots:1.000000
      ::::::::
      3129:1:RUNNING:1077903480:31:G:global:license:1.000000
      3129:1:RUNNING:1077903480:31:Q:all.q@carc:slots:1.000000

Each section shows, for a schedule interval, all resource usage that was taken into account. RUNNING entries show usage of jobs that were already running at the start of the interval. STARTING entries show the immediate uses that were decided within the interval. RESERVING entries show uses that are planned for the future, that is, reservations.

The format of the schedule file is as follows:

• jobID – The job ID
• taskID – The array task ID, or 1 in the case of nonarray jobs
• state – Can be RUNNING, SUSPENDED, MIGRATING, STARTING, or RESERVING
• start-time – Start time in seconds after 1.1.1070
• duration – Assumed job duration in seconds
• level-char – Can be P (for parallel environment), G (for global), H (for host), or Q (for queue)
• object-name – The name of the parallel environment, host, or queue
• resource-name – The name of the consumable resource
• usage – The resource usage incurred by the job

The line :::::::: marks the beginning of a new schedule interval.

Note The schedule file is not truncated. Be sure to turn monitoring off if you do not have an automated procedure that is set up to truncate the file.

What Happens in a Scheduler Interval

The Scheduler schedules work in intervals. Between scheduling actions, the Grid Engine system keeps information about significant events such as the following:

• Job submission
• Job completion
• Job cancellation
• An update of the cluster configuration
• Registration of a new machine in the cluster

When scheduling occurs, the scheduler first does the following:

• Takes into account all significant events
• Sorts jobs and queues according to the administrator's specifications
• Takes into account all the jobs' resource requirements
• Reserves resources for jobs in a forward-looking schedule

Then the Grid Engine system does the following tasks, as needed:

• Dispatches new jobs
• Suspends running jobs
• Increases or decreases the resources allocated to running jobs
• Maintains the status quo

If share-based scheduling is used, the calculation takes into account the usage that has already occurred for that user or project.

If scheduling is not at least in part share-based, the calculation ranks all the jobs running and waiting to run. The calculation then takes the most important job until the resources in the cluster (CPU, memory, and I/O bandwidth) are used as fully as possible.

Scheduler Monitoring

If the reasons why a job does not get started are unclear to you, run the qalter -w v command for the job. The Grid Engine software assumes an empty cluster and checks whether any queue that is suitable for the job is available.

Further information can be obtained by running the qstat -j job-id command. This command prints a summary of the job's request profile. The summary also includes the reasons why the job was not scheduled in the last scheduling interval. Running the qstat -j command without a job ID summarizes the reasons for all jobs not being scheduled in the last scheduling interval.

Note Collection of job scheduling information must be switched on in the scheduler configuration sched_conf(5). Refer to the schedd_job_info parameter description in the sched_conf(5) man page, or to Changing the Scheduler Configuration With QMON.

To retrieve even more detail about the decisions of the scheduler sge_schedd, use the -tsm option of the qconf command. This command forces sge_schedd to write trace output to the file.

Configuring the Scheduler

Refer to Configuring Policy-Based Resource Management With QMON for details on the scheduling administration of resource-sharing policies of the Grid Engine system. The following sections focus on administering the scheduler configuration sched_conf and related issues.

Default Scheduling

The default scheduling is a first-in, first-out policy. In other words, the first job that is submitted is the first job the scheduler examines to dispatch it to a queue. If the first job in the list of pending jobs finds a queue that is suitable and available, that job is started first. A job ranked behind the first job can be started first only if the first job fails to find a suitable free resource.

The default strategy is to select queue instances on the least-loaded host, provided that the queues deliver suitable service for the job's resource requirements. If several suitable queues share the same load, the queue to be selected is unpredictable.

Scheduling Alternatives

You can modify the job scheduling and queue selection strategy in various ways:

• Changing the scheduling algorithm
• Scaling system load
• Selecting queue by sequence number
• Selecting queue by share
• Restricting the number of jobs per user or per group

The following sections explore these alternatives in detail.

Changing the Scheduling Algorithm

The scheduler configuration parameter algorithm provides a selection for the scheduling algorithm in use. See the sched_conf(5) man page for further information. Currently, default is the only allowed setting.

Scaling System Load

To select the queue to run a job, the Grid Engine system uses the system load information on the machines that host queue instances. This queue selection scheme builds up a load-balanced situation, thus guaranteeing better use of the available resources in a cluster.

However, the system load may not always tell the truth. For example, if a multi-CPU machine is compared to a single CPU system, the multiprocessor system usually reports higher load figures, because it probably runs more processes. The system load is a measurement strongly influenced by the number of processes trying to get CPU access. But multi-CPU systems are capable of satisfying a much higher load than single-CPU machines. This problem is addressed by processor-number-adjusted sets of load values that are reported by default by sge_execd. Use these load parameters instead of the raw load values to avoid the problem described earlier. See Load Parameters and the $SGE_ROOT/doc/load_parameters.asc file for details.

Another example of potentially improper interpretation of load values is when systems have marked differences in their performance potential or in their price performance ratio. In both cases, equal load values do not mean that arbitrary hosts can be selected to run a job. In this situation, the administrator should define load scaling factors for the relevant execution hosts and load parameters. See Configuring Execution Hosts With QMON, and related sections.

Note The scaled load parameters are also compared against the load threshold lists load-thresholds and migr-load-thresholds. See the queue_conf(5) man page for details.

Another problem associated with load parameters is the need for an application-dependent and site-dependent interpretation of the values and their relative importance. The CPU load might be dominant for a certain type of application that is common at a particular site. By contrast, the memory load might be more important for another site and for the application profile to which the site's compute cluster is dedicated. To address this problem, the Grid Engine system enables the administrator to specify a load formula in the scheduler configuration file sched_conf. See the sched_conf(5) man page for more details. Site-specific information on resource usage and capacity planning can be taken into account by using site-defined load parameters and consumable resources in the load formula. See the sections Adding Site-Specific Load Parameters and Consumable Resources.

Finally, the time dependency of load parameters must be taken into account. The load that is imposed by the jobs that are running on a system varies in time. Often the load, for example, the CPU load, requires some amount of time to be reported in the appropriate quantity by the operating system. If a job recently started, the reported load might not provide an accurate representation of the load that the job has imposed on that host. The reported load adapts to the real load over time. But the period of time in which the reported load is too low might lead to an over-subscription of that host. The Grid Engine system enables the administrator to specify load adjustment factors that are used in the scheduler to compensate for this problem. See the sched_conf(5) man page for detailed information on how to set these load adjustment factors.

Load adjustments are used to virtually increase the measured load after a job is dispatched. In the case of oversubscribed machines, this helps to align with load thresholds. If you do not need load adjustments, you should turn them off. Load adjustments impose additional work on the scheduler in connection with sorting hosts and load thresholds verification.

To disable load adjustments, on the Load Adjustment tab of the Scheduler Configuration dialog box, set the Decay Time to zero, and delete all load adjustment values in the table. See Changing the Scheduler Configuration With QMON.

Selecting Queue by Sequence Number

Another way to change the default scheme for queue selection is to set the global cluster configuration parameter queue_sort_method to seq_no instead of to the default load. In this case, the system load is no longer used as the primary method to select queues. Instead, the sequence numbers that are assigned to the queues by the queue configuration parameter seq_no define a fixed order for queue selection. The queues must be suitable for the considered job, and they must be available. See the queue_conf(5) and sched_conf(5) man pages for more details.

This queue selection policy is useful if the machines that offer batch services at your site are ranked in a monotonous price per job order. For example, a job running on Machine A costs 1-unit of money. The same job costs 10-units on Machine B. And on Machine C the job costs 100-units. Thus the preferred scheduling policy is to first fill up Host A and then to use Host B. Host C is used only if no alternative remains.

Note If you have changed the method of queue selection to seq_no, and the considered queues all share the same sequence number, queues are selected by the default load.

Selecting Queue by Share

The goal of this method is to place jobs so as to attempt to meet the targeted share of global system resources for each job. This method takes into account the resource capability represented by each host in relation to all the system resources. This method tries to balance the percentage of tickets for each host (that is, the sum of tickets for all jobs running on a host) with the percentage of the resource capability that particular host represents for the system. See Configuring Execution Hosts With QMON for instructions on how to define the capacity of a host.

The host's load, although of secondary importance, is also taken into account in the sorting. Choose this sorting method for a site that uses the share-tree policy.

Restricting the Number of Jobs per User or Group

The administrator can assign an upper limit to the number of jobs that any user or any UNIX group can run at any time. To enforce this feature, do one of the following:

• Set maxujobs as described in the sched_conf(5) man page.
• On the General Parameters tab of the Scheduler Configuration dialog box, use the Max Jobs/User field to set the maximum number of jobs a user or user group can run concurrently.

Changing the Scheduler Configuration With QMON

On the QMON Main Control window, click the Scheduler Configuration button.

The Scheduler Configuration dialog box appears. The dialog box has two tabs:

• General Parameters tab
• Load Adjustment tab

To change general scheduling parameters, click the General Parameters tab. The General Parameters tab looks like the following figure.

Use the General Parameters tab to set the following parameters:

• Algorithm – The scheduling algorithm. See Changing the Scheduling Algorithm.
• Schedule Interval – The regular time interval between scheduler runs.
• Reprioritize Interval – The regular time interval to reprioritize jobs on the execution hosts, based on the current ticket amount for running jobs. To turn reprioritizing off, set this parameter to zero.
• Max Jobs/User – The maximum number of jobs that are allowed to run concurrently per user and per UNIX group. See Restricting the Number of Jobs per User or Group.
• Sort by – The queue sorting scheme, either sorting by load or sorting by sequence number. See Selecting Queue by Sequence Number.
• Job Scheduling Information – Whether job scheduling information is accessible through qstat -j, or whether this information should be collected only for a range of job IDs. You should turn on general collection of job scheduling information only temporarily, in case an extremely high number of jobs are pending.

Scheduler monitoring can help you find out the reason why certain jobs are not dispatched. However, providing this information for all jobs at all times can consume resources. Such information is usually not needed.

• Load Formula – The load formula to use to sort hosts and queues.
• Flush Submit Seconds – The number of seconds that the scheduler waits after a job is submitted before the scheduler is triggered. To disable the flush after a job is submitted, set this parameter to zero.
• Flush Finish Seconds – The number of seconds that the scheduler waits after a job has finished before the scheduler is triggered. To disable the flush after a job has finished, set this parameter to zero.
• Maximum Reservation – The maximum number of resource reservations that can be scheduled within a scheduling interval. See Resource Reservation and Backfilling.
• Params – Use this setting to specify additional parameters to pass to the scheduler. Params can be PROFILE or MONITOR. If you set PROFILE=1, the scheduler will log profiling information that summarizes each scheduling run. If you set MONITOR=1, the scheduler will record information for each scheduling run in the file $SGE_ROOT/$SGE_CELL/common/schedule.
  By default, the Grid Engine system schedules job runs in a fixed schedule interval. You can use the Flush Submit Seconds and Flush Finish Seconds parameters to configure immediate scheduling. For more information, see Immediate Scheduling.

To change load adjustment parameters, click the Load Adjustment tab. The Load Adjustment tab looks like the following figure:

The Load Adjustment tab displays following parameters:

• Decay Time – The decay time for the load adjustment.
• A table of load adjustment values listing all load and consumable attributes for which an adjustment value is currently defined.

To change load adjustment parameters, do the following:

• To add load values to the list, click the Load or the Value column heading. A selection list appears with all resource attributes that are attached to the hosts.
• To add a resource attribute in the Load column of the Consumable/Fixed Attributes table, use the Attribute Selection dialog box to select one of the attributes, and then click OK.
• To modify an existing value, double-click the Value field.
• To delete a resource attribute, select it, and then press Control-D or click mouse button 3. A dialog box asks you to confirm the deletion.

See Scaling System Load for background information. See the sched_conf(5) man page for more details about the scheduler configuration.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Managing Policies

Grid Engine policies are implemented in conjunction with the Grid Engine scheduler. For information, see Managing the Scheduler.

About Grid Engine Policies

The Grid Engine software orchestrates the delivery of computational power, based on enterprise resource policies that the administrator manages. The software uses these policies to examine available computer resources in the grid. The software gathers these resources, and then it allocates and delivers them automatically, in a way that optimizes usage across the grid.

To enable cooperation in the grid, project owners must do the following:

• Negotiate policies
• Ensure that policies for manual overrides for unique project requirements are flexible
• Automatically monitor and enforce policies

As administrator, you can define high-level usage policies that are customized for your site. Four such policies are available:

• Urgency policy – See Configuring the Urgency Policy
• Share-based policy – See Configuring the Share-Based Policy
• Functional policy – See Configuring the Functional Policy
• Override policy – See Configuring the Override Policy

Policy management automatically controls the use of shared resources in the cluster to achieve your goals. High-priority jobs are dispatched preferentially. These jobs receive greater CPU entitlements when they are competing with other, lower-priority jobs. The Grid Engine software monitors the progress of all jobs. It adjusts their relative priorities correspondingly, and with respect to the goals that you define in the policies.

This policy-based resource allocation grants each user, team, department, and all projects an allocated share of system resources. This allocation of resources extends over a specified period of time, such as a week, a month, or a quarter.

Configuring Policy-Based Resource Management With QMON

On the QMON Main Control window, click the Policy Configuration button. The Policy Configuration dialog box appears.

The Policy Configuration dialog box lets you directly edit the following information:

• Policy Importance Factor – Specifies the relative importance of the priority types that control the sorting order of jobs. See Specifying Policy Priority.
• Urgency Policy – Defines an urgency value for each job. See Configuring the Urgency Policy.
• Ticket Policy – Lets you modify the number of tickets that are allocated to the share tree policy and the functional policy. See Configuring Ticket-Based Policies.

You can also access detailed configuration dialog boxes for the three ticket-based policies.

• Share-based policy – See Configuring the Share-Based Policy
• Functional policy – See Configuring the Functional Policy
• Override policy – See Configuring the Override Policy

To refresh the information displayed in the Policy Configuration dialog box, click Refresh.

To save any changes that you make to the Policy Configuration, click Apply. To close the dialog box without saving changes, click Done.

Specifying Policy Priority

Before the Grid Engine system dispatches jobs, the jobs are brought into priority order, highest priority first. Without any administrator influence, the order is first-in, first-out (FIFO).

On the Policy Configuration dialog box, under Policy Importance Factor, you can specify the relative importance of the three priority types that control the sorting order of jobs. For example, if you specify Priority as 1, Urgency as 0.1, and Ticket as 0.01, job priority that is specified by the qsub --p command is given the most weight, job priority that is specified by the Urgency Policy is considered next, and job priority that is specified by the Ticket Policy is given the least weight.

• Priority – Also called POSIX priority. The --p option of the qsub command specifies site-specific priority policies.
• Urgency Policy – Jobs can have an urgency value that determines their relative importance. Pending jobs are sorted according to their urgency value.
• Ticket Policy – Jobs are always treated according to their relative importance as defined by the number of tickets that the jobs have. Pending jobs are sorted in ticket order.

For more information about job priorities, see Job Sorting.

You can specify a weighting factor for each priority type. This weighting factor determines the degree to which each type of priority affects overall job priority. To make it easier to control the range of values for each priority type, normalized values are used instead of the raw ticket values, urgency values, and POSIX priority values.

The following formula expresses how a job's priority values are determined:

Job priority = Urgency * normalized urgency value + 
Ticket * normalized ticket value + 
Priority * normalized priority value

Configuring the Urgency Policy

The Urgency Policy defines an urgency value for each job. This urgency value is determined by the sum of the following three contributing elements:

• Resource requirement – Each resource attribute defined in the complex can have an urgency value. For information about the setting urgency values for resource attributes, see Configuring Complex Resource Attributes With QMON. Each job request for a resource attribute adds the attribute's urgency value to the total.
• Deadline – The urgency value for deadline jobs is determined by dividing the Weight Deadline specified in the Policy Configuration dialog box by the free time, in seconds, until the job's deadline initiation time specified by the qsub --dl command.
• Waiting time – The urgency value for a job's waiting time is determined by multiplying the job's waiting time by the Weight Waiting Time specified in the Policy Configuration dialog box. The job's waiting time is measured in seconds.

For details about how the Grid Engine system arrives at the urgency value total, see About the Urgency Policy.

Configuring Ticket-Based Policies

The tickets that are currently assigned to individual policies are listed under Current Active Tickets in the Policy Configuration dialog box. The numbers reflect the relative importance of the policies. The numbers indicate whether a certain policy currently dominates the cluster or whether policies are in balance.

Tickets provide a quantitative measure. For example, you might assign twice as many tickets to the share-based policy as you assign to the functional policy. This means that twice the resource entitlement is allocated to the share-based policy than is allocated to the functional policy. In this sense, tickets behave very much like stock shares.

The total number of all tickets has no particular meaning. Only the relations between policies counts. Hence, total ticket numbers are usually quite high to allow for fine adjustment of the relative importance of the policies.

Under Edit Tickets, you can modify the number of tickets that are allocated to the share tree policy and the functional policy. For details, see Editing Tickets.

Select the Share Override Tickets check box to control the total ticket amount distributed by the override policy. Deselect the Share Override Tickets check box to control the importance of individual jobs relative to the ticket pools that are available for the other policies and override categories. With this setting, the number of jobs that are under a category member does not matter. The jobs always get the same number of tickets. However, the total number of override tickets in the system increases as the number of jobs with a right to receive override tickets increases. Other policies can lose importance in such cases. For detailed information, see Sharing Override Tickets.

Select the Share Functional Tickets check box to give a category member a constant entitlement level for the sum of all its jobs. Deselect the check box to give each job the same entitlement level, based on its category member's entitlement. For detailed information, see Sharing Functional Ticket Shares.

You can set the maximum number of jobs that can be scheduled in the functional policy. The default value is 200.

You can set the maximum number of pending subtasks that are allowed for each array job. The default value is 50. Use this setting to reduce scheduling overhead.

You can specify the Ticket Policy Hierarchy to resolve certain cases of conflicting policies. The resolving of policy conflicts applies particularly to pending jobs. For detailed information, see Setting the Ticket Policy Hierarchy.

Editing Tickets

You can edit the total number of share-tree tickets and functional tickets. Override tickets are assigned directly through the override policy configuration. The other ticket pools are distributed automatically among jobs that are associated with the policies and with respect to the actual policy configuration.

Note All share-based tickets and functional tickets are always distributed among the jobs associated with these policies. Override tickets might not be applicable to the currently active jobs. Consequently, the active override tickets might be zero, even though the override policy has tickets defined.

Sharing Override Tickets

The administrator assigns tickets to the different members of the override categories, that is, to individual users, projects, departments, or jobs. Consequently, the number of tickets that are assigned to a category member determines how many tickets are assigned to jobs under that category member. For example, the number of tickets that are assigned to user A determines how many tickets are assigned to all jobs of user A.

Note The number of tickets that are assigned to the job category does not determine how many tickets are assigned to jobs in that category.

Use the Share Override Tickets check box to set the share_override_tickets parameter of sched_conf(5). This parameter controls how job ticket values are derived from their category member ticket value. When you select the Share Override Tickets check box, the tickets of the category members are distributed evenly among the jobs under this member. If you deselect the Share Override Tickets check box, each job inherits the ticket amount defined for its category member. In other words, the category member tickets are replicated for all jobs underneath.

Select the Share Override Tickets check box to control the total ticket amount distributed by the override policy. With this setting, ticket amounts that are assigned to a job can become negligibly small if many jobs are under one category member. For example, ticket amounts might diminish if many jobs belong to one member of the user category.

Deselect the Share Override Tickets check box to control the importance of individual jobs relative to the ticket pools that are available for the other policies and override categories. With this setting, the number of jobs that are under a category member does not matter. The jobs always get the same number of tickets. However, the total number of override tickets in the system increases as the number of jobs with a right to receive override tickets increases. Other policies can lose importance in such cases.

Sharing Functional Ticket Shares

The functional policy defines entitlement shares for the functional categories. Then the policy defines shares for all members of each of these categories. The functional policy is thus similar to a two-level share tree. The difference is that a job can be associated with several categories at the same time. The job belongs to a particular user, for instance, but the job can also belong to a project or a department.

However, as in the share tree, the entitlement shares that a job receives from a functional category is determined by the following:

• The shares that are defined for its corresponding category member (for example, its project)
• The shares that are given to the category (project instead of user, department, and so on)

Use the Share Functional Tickets check box to set the share_functional_shares parameter of sched_conf(5). This parameter defines how the category member shares are used to determine the shares of a job. The shares assigned to the category members, such as a particular user or project, can be replicated for each job. Alternatively, shares can be distributed among the jobs under the category member.

• Selecting the Share Functional Tickets check box means that functional shares are replicated among jobs.
• Deselecting the Share Functional Tickets check box means that functional shares are distributed among jobs.

Those shares are comparable to stock shares. Such shares have no effect for the jobs that belong to the same category member. All jobs under the same category member have the same number of shares in both cases. But the share number has an effect when comparing the share amounts within the same category. Jobs with many siblings that belong to the same category member receive relatively small share portions if you select the Share Functional Tickets check box. On the other hand, if you clear the Share Functional Tickets check box, all sibling jobs receive the same share amount as their category member.

Select the Share Functional Tickets check box to give a category member a constant entitlement level for the sum of all its jobs. The entitlement of an individual job can get negligibly small, however, if the job has many siblings.

Deselect the Share Functional Tickets check box to give each job the same entitlement level, based on its category member's entitlement. The number of job siblings in the system does not matter.

Note A category member with many jobs underneath can dominate the functional policy.

Be aware that the setting of share functional shares does not determine the total number of functional tickets that are distributed. The total number is always as defined by the administrator for the functional policy ticket pool. The share functional shares parameter influences only how functional tickets are distributed within the functional policy.

Example – Functional Policy

The following example describes a common scenario where a user wishes to translate the Sun Grid Engine 5.3 Scheduler Option -user_sort true to a Sun Grid Engine 6.2 configuration but does not understand the share override functional policy ticket feature.

For a plain user-based equal share, you configure your global configuration sge_conf(5) with

Then you use -weight_tickets_functional 10000 in the scheduler configuration sched_conf(5). This action causes the functional policy to be used for user-based equal share scheduling with 100 shares for each user.

Tuning Scheduling Run Time

Pending jobs are sorted according to the number of tickets that each job has, as described in Job Sorting. The scheduler reports the number of tickets each pending job has to the master daemon sge_qmaster. However, on systems with very large numbers of jobs, you might want to turn off ticket reporting. When you turn off ticket reporting, you disable ticket-based job priority. The sort order of jobs is based only on the time each job is submitted.

To turn off the reporting of pending job tickets to sge_qmaster, clear the Report Pending Job Tickets check box on the Policy Configuration dialog box. Doing so sets the report_pjob_tickets parameter of sched_conf(5) to false.

Setting the Ticket Policy Hierarchy

Ticket policy hierarchy provides the means to resolve certain cases of conflicting ticket policies. The resolving of ticket policy conflicts applies particularly to pending jobs.

Such cases can occur in combination with the share-based policy and the functional policy. With both policies, assigning priorities to jobs that belong to the same leaf-level entities is done on a first-come, first-served basis. Leaf-level entities include the following:

• User leaves in the share tree
• Project leaves in the share tree
• Any member of the following categories in the functional policy: user, project, department, or queue

Members of the job category are not included among leaf-level entities. So, for example, the first job of the same user gets the most, the second gets the next most, the third next, and so on.

A conflict can occur if another policy mandates an order that is different. So, for example, the override policy might define the third job as the most important, whereas the first job that is submitted should come last.

A policy hierarchy might gives the override policy higher priority over the share-tree policy or the functional policy. Such a policy hierarchy ensures that high-priority jobs under the override policy get more entitlements than jobs in the other two policies. Such jobs must belong to the same leaf level entity (user or project) in the share tree.

The Ticket Policy Hierarchy can be a combination of up to three letters. These letters are the first letters of the names of the following three ticket policies:

• S – Share-based
• F – Functional
• O – Override

Use these letters to establish a hierarchy of ticket policies. The first letter defines the top policy. The last letter defines the bottom of the hierarchy. Policies that are not listed in the policy hierarchy do not influence the hierarchy. However, policies that are not listed in the hierarchy can still be a source for tickets of jobs. However, those tickets do not influence the ticket calculations in other policies. All tickets of all policies are added up for each job to define its overall entitlement.

The following examples describe two settings and how they influence the order of the pending jobs:

• Override, Share-Based (OS) Hierarchy Setting

  policy_hierarchy=OS
  

  1. The override policy assigns the appropriate number of tickets to each pending job.
  2. The number of tickets determines the entitlement assignment in the share tree in case two jobs belong to the same user or to the same leaf-level project. Then the share tree tickets are calculated for the pending jobs.
  3. The tickets from the override policy and from the share-tree policy are added together, along with all other active policies not in the hierarchy. The job with the highest resulting number of tickets has the highest entitlement.
• Override, Functional (OF) Hierarchy Setting

  policy_hierarchy=OF
  

  1. The override policy assigns the appropriate number of tickets to each pending job. Then the tickets from the override policy are added up.
  2. The resulting number of tickets influences the entitlement assignment in the functional policy in case two jobs belong to the same functional category member. Based on this entitlement assignment, the functional tickets are calculated for the pending jobs.
  3. The resulting value is added to the ticket amount from the override policy. The job with the highest resulting number of tickets has the highest entitlement.

All combinations of the three letters are theoretically possible, but only a subset of the combinations are meaningful or have practical relevance. The last letter should always be S or F, because only those two policies can be influenced due to their characteristics described in the examples.

The following form is recommended for policy_hierarchy settings:

[O][S|F]

If the override policy is present, O should occur as the first letter only, because the override policy can only influence. The share-based policy and the functional policy can only be influenced. Therefore S or F should occur as the last letter.

Configuring the Share-Based Policy

Share-based scheduling grants each user and project its allocated share of system resources during an accumulation period such as a week, a month, or a quarter. Share-based scheduling is also called share tree scheduling. It constantly adjusts each user's and project's potential resource share for the near term, until the next scheduling interval. Share-based scheduling is defined for user or for project, or for both.

Share-based scheduling ensures that a defined share is guaranteed to the instances that are configured in the share tree over time. Jobs that are associated with share-tree branches where fewer resources were consumed in the past than anticipated are preferred when the system dispatches jobs. At the same time, full resource usage is guaranteed, because unused share proportions are still available for pending jobs associated with other share-tree branches.

By giving each user or project its targeted share as far as possible, groups of users or projects also get their targeted share. Departments or divisions are examples of such groups. Fair share for all entities is attainable only when every entity that is entitled to resources contends for those resources during the accumulation period. If a user, a project, or a group does not submit jobs during a given period, the resources are shared among those who do submit jobs.

Share-based scheduling is a feedback scheme. The share of the system to which any user or user-group, or project or project-group, is entitled is a configuration parameter. The share of the system to which any job is entitled is based on the following factors:

• The share allocated to the job's user or project
• The accumulated past usage for each user and user group, and for each project and project group. This usage is adjusted by a decay factor. "Old" usage has less impact.

The Grid Engine software keeps track of how much usage users and projects have already received. At each scheduling interval, the Scheduler adjusts all jobs' share of resources. Doing so ensures that all users, user groups, projects, and project groups get close to their fair share of the system during the accumulation period. In other words, resources are granted or are denied to keep everyone more or less at their targeted share of usage.

The Half-Life Factor

Half-life is how fast the system "forgets" about a user's resource consumption. The administrator decides whether to penalize a user for high resource consumption, be it six months ago or six days ago. The administrator also decides how to apply the penalty. On each node of the share tree, Grid Engine software maintains a record of users' resource consumption.

With this record, the system administrator can decide how far to look back to determine a user's under-usage or over-usage when setting up a share-based policy. The resource usage in this context is the mathematical sum of all the computer resources that are consumed over a "sliding window of time."

The length of this window is determined by a "half-life" factor, which in the Grid Engine system is an internal decay function. This decay function reduces the impact of accrued resource consumption over time. A short half-life quickly lessens the impact of resource overconsumption. A longer half-life gradually lessens the impact of resource overconsumption.

This half-life decay function is a specified unit of time. For example, consider a half-life of seven days that is applied to a resource consumption of 1,000 units. This half-life decay factor results in the following usage "penalty" adjustment over time:

• 500 after 7 days
• 250 after 14 days
• 125 after 21 days
• 62.5 after 28 days

The half-life-based decay diminishes the impact of a user's resource consumption over time, until the effect of the penalty is negligible.

Note Override tickets that a user receives are not subjected to a past usage penalty, because override tickets belong to a different policy system. The decay function is a characteristic of the share-tree policy only.

Compensation Factor

Sometimes the comparison shows that actual usage is well below targeted usage. In such a case, the adjusting of a user's share or a project's share of resource can allow a user to dominate the system. Such an adjustment is based on the goal of reaching target share. This domination might not be desirable.

The compensation factor enables an administrator to limit how much a user or a project can dominate the resources in the near term.

For example, a compensation factor of two limits a user's or project's current share to twice its targeted share. Assume that a user or a project should get 20 percent of the system resources over the accumulation period. If the user or project currently gets much less, the maximum that it can get in the near term is only 40 percent.

The share-based policy defines long-term resource entitlements of users or projects as per the share tree. When combined with the share-based policy, the compensation factor makes automatic adjustments in entitlements.

If a user or project is either under or over the defined target entitlement, the Grid Engine system compensates. The system raises or lowers that user's or project's entitlement for a short term over or under the long-term target. This compensation is calculated by a share tree algorithm.

The compensation factor provides an additional mechanism to control the amount of compensation that the Grid Engine system assigns. The additional compensation factor (CF) calculation is carried out only if the following conditions are true:

• Short-term-entitlement is greater than long-term-entitlement multiplied by the CF.
• The CF is greater than 0.

If either condition is not true, or if both conditions are not true, the compensation as defined and implemented by the share-tree algorithm is used.

The smaller the value of the CF, the greater is its effect. If the value is greater than 1, the Grid Engine system's compensation is limited. The upper limit for compensation is calculated as long-term-entitlement multiplied by the CF. And as defined earlier, the short-term entitlement must exceed this limit before anything happens based on the compensation factor.

If the CF is 1, the Grid Engine system compensates in the same way as with the raw share-tree algorithm. So a value of one has an effect that is similar to a value of zero. The only difference is an implementation detail. If the CF is one, the CF calculations are carried out without an effect. If the CF is zero, the calculations are suppressed.

If the value is less than 1, the Grid Engine system overcompensates. Jobs receive much more compensation than they are entitled to based on the share-tree algorithm. Jobs also receive this overcompensation earlier, because the criterion for activating the compensation is met at lower short-term entitlement values. The activating criterion is short-term-entitlement > long-term-entitlement * CF.

Hierarchical Share Tree

The share-based policy is implemented through a hierarchical share tree. The share tree specifies, for a moving accumulation period, how system resources are to be shared among all users and projects. The length of the accumulation period is determined by a configurable decay constant. The Grid Engine system bases a job's share entitlement on the degree to which each parent node in the share tree reaches its accumulation limit. A job's share entitlement is based on its leaf node share allocation, which in turn depends on the allocations of its parent nodes. All jobs associated with a leaf node split the associated shares.

The entitlement derived from the share tree is combined with other entitlements, such as entitlements from a functional policy, to determine a job's net entitlement. The share tree is allotted the total number of tickets for share-based scheduling. This number determines the weight of share-based scheduling among the four scheduling policies.

The share tree is defined during installation. The share tree can be altered at any time. When the share tree is edited, the new share allocations take effect at the next scheduling interval.

Configuring the Share-Tree Policy With QMON

On the QMON Policy Configuration dialog box, click Share Tree Policy. The Share Tree Policy dialog box appears.

Node Attributes

Under Node Attributes, the attributes of the selected node are displayed:

• Identifier – A user, project, or agglomeration name.
• Shares – The number of shares that are allocated to this user or project.

  Note Shares define relative importance. They are not percentages. Shares also do not have quantitative meaning. The specification of hundreds or even thousands of shares is generally a good idea, as high numbers allow fine tuning of importance relationships.

• Level Percentage – This node's portion of the total shares at the level of the same parent node in the tree. The number of this node's shares divided by the sum of its and its sibling's shares.
• Total Percentage – This node's portion of the total shares in the entire share tree. The long-term targeted resource share of the node.
• Actual Resource Usage – The percentage of all the resources in the system that this node has consumed so far in the accumulation period. The percentage is expressed in relation to all nodes in the share tree.
• Targeted Resource Usage – Same as Actual Resource Usage, but only taking the currently active nodes in the share tree into account. Active nodes have jobs in the system. In the short term, the Grid Engine system attempts to balance the entitlement among active nodes.
• Combined Usage – The total usage for the node. Combined Usage is the sum of the usage that is accumulated at this node. Leaf nodes accumulate the usage of all jobs that run under them. Inner nodes accumulate the usage of all descendant nodes. Combined Usage includes CPU, memory, and I/O usage according to the ratio specified under Share Tree Policy Parameters. Combined usage is decayed at the half-life decay rate that is specified by the parameters.

When a user node or a project node is removed and then added back, the user's or project's usage is retained. A node can be added back either at the same place or at a different place in the share tree. You can zero out that usage before you add the node back to the share tree. To do so, first remove the node from the users or projects configured in the Grid Engine system. Then add the node back to the users or projects there.

Users or projects that were not in the share tree but that ran jobs have nonzero usage when added to the share tree. To zero out usage when you add such users or projects to the tree, first remove them from the users or projects configured in the Grid Engine system. Then add them to the tree.

To add an interior node under the selected node, click Add Node. A blank Node Info window appears, where you can enter the node's name and number of shares. You can enter any node name or share number.

To add a leaf node under the selected node, click Add Leaf. A blank Node Info window appears, where you can enter the node's name and number of shares. The node's name must be an existing Grid Engine user (Configuring User Objects With QMON) or project (Defining Projects).

The following rules apply when you are adding a leaf node:

• All nodes have a unique path in share tree.
• A project is not referenced more than once in a share tree.
• A user appears only once in a project subtree.
• A user appears only once outside of a project subtree.
• A user does not appear as a nonleaf node.
• All leaf nodes in a project subtree reference a known user or the reserved name default. See a detailed description of this special user in About the Special User default.
• Project subtrees do not have sub-projects.
• All leaf nodes not in a project subtree reference a known user or known project.
• All user leaf nodes in a project subtree have access to the project.

To edit the selected node, click Modify. A Node Info window appears. The window displays the mode's name and its number of shares.

To cut or copy the selected node to a buffer, click Cut or Copy. To paste under the selected node the contents of the most recently cut or copied node, click Paste.

To delete the selected node and all its descendants, click Delete.

To clear the entire share-tree hierarchy, click Clear Usage. Clear the hierarchy when the share-based policy is aligned to a budget and needs to start from scratch at the beginning of each budget term. The Clear Usage facility also is handy when setting up or modifying test Grid Engine software environments.

QMON periodically updates the information displayed in the Share Tree Policy dialog box. Click Refresh to force the display to refresh immediately.

To save all the node changes that you make, click Apply. To close the dialog box without saving changes, click Done.

To search the share tree for a node name, click Find, and then type a search string. Node names are indicated which begin with the case sensitive search string. Click Find Next to find the next occurrence of the search string.

Click Help to open the online help system.

Share Tree Policy Parameters

To display the Share Tree Policy Parameters, click the arrow at the right of the Node Attributes.

• CPU [%] slider — This slider's setting indicates what the percentage of Combined Usage CPU is. When you change this slider, the MEM and I/O sliders change to compensate for the change in CPU percentage.
• MEM [%] slider — This slider's setting indicates what the percentage of Combined Usage memory is. When you change this slider, the CPU and I/O sliders change to compensate for the change in MEM percentage.
• I/O [%] slider — This slider's setting indicates what the percentage of Combined Usage I/O is. When you change this slider, the CPU and MEM sliders change to compensate for the change in I/O percentage.

  Note CPU %, MEM %, and I/O % always adds up to 100%.

• Lock Symbol — When a lock is open, the slider that it guards can change freely. The slider can change either because the slider was moved or because it is compensating for another slider's being moved. When a lock is closed, the slider that it guards cannot change. If two locks are closed and one lock is open, no sliders can be changed.
• Half-life — Use this field to specify the half-life for usage. Usage is decayed during each scheduling interval so that any particular contribution to accumulated usage has half the value after a duration of half-life.
• Days/Hours selection menu — Select whether half-life is to be measured in days or hours.
• Compensation Factor — This field accepts a positive integer for the compensation factor. Reasonable values range between 2 and 10.

The actual usage of a user or project can be far below its targeted usage. The compensation factor prevents such users or projects from dominating resources when they first get those resources. See Compensation Factor for more information.

About the Special User default

You can use the special user default to reduce the amount of share-tree maintenance for sites with many users. Under the share-tree policy, a job's priority is determined based on the node that the job maps to in the share tree. Users who are not explicitly named in the share tree are mapped to the default node, if it exists.

The specification of a single default node allows for a simple share tree to be created. Such a share tree makes user-based fair sharing possible.

You can use the default user also in cases where the same share entitlement is assigned to most users. Same share entitlement is also known as equal share scheduling.

The default user configures all user entries under the default node, giving the same share amount to each user. Each user who submits jobs receives the same share entitlement as that configured for the default user. To activate the facility for a particular user, you must add this user to the list of Grid Engine users.

The share tree displays "virtual" nodes for all users who are mapped to the default node. The display of virtual nodes enables you to examine the usage and the fair-share scheduling parameters for users who are mapped to the default node.

You can also use the default user for "hybrid" share trees, where users are subordinated under projects in the share tree. The default user can be a leaf node under a project node.

The short-term entitlements of users vary according to differences in the amount of resources that the users consume. However, long-term entitlements of users remain the same.

You might want to assign lower or higher entitlements to some users while maintaining the same long-term entitlement for all other users. To do so, configure a share tree with individual user entries next to the default user for those users with special entitlements.

In Example A, all users submitting to Project A get equal long-term entitlements. The users submitting to Project B only contribute to the accumulated resource consumption of Project B. Entitlements of Project B users are not managed.

Example A

Compare Example A with Example B:

Example B

In Example B, treatment for Project A is the same as for Example A. But all default users who submit jobs to Project B, except users A and B, receive equal long-term resource entitlements. Default users have 20 shares. User A, with 10 shares, receives half the entitlement of the default users. User B, with 40 shares, receives twice the entitlement as the default users.

How to Create Project-Based Share-Tree Scheduling

The objective of this setup is to guarantee a certain share assignment of all the cluster resources to different projects over time.

1. Specify the number of share-tree tickets (for example, 1000000) in the scheduler configuration.
   See Configuring Policy-Based Resource Management With QMON, and the sched_conf(5) man page.
   
   
2. (Optional) Add one user for each scheduling-relevant user.
   See Configuring User Objects With QMON, and the user(5) man page.
   
   
3. Add one project for each scheduling-relevant project.
   See Defining Projects With QMON, and the project(5) man page.
   
   
4. Use QMON to set up a share tree that reflects the structure of all scheduling-relevant projects as nodes.
   See Configuring the Share-Tree Policy With QMON.
   
   
5. Assign share-tree shares to the projects.
   For example, if you are creating project-based share-tree scheduling with first-come, first-served scheduling among jobs of the same project, a simple structure might look like the following:
   
   If you are creating project-based share-tree scheduling with equal shares for each user, a simple structure might look like the following:
   
   If you are creating project-based share-tree scheduling with individual user shares in each project, add users as leaves to their projects. Then assign individual shares. A simple structure might look like the following:
   
   If you want to assign individual shares to only a few users, designate the user default in combination with individual users below a project node. For example, you can condense the tree illustrated previously into the following:
   

Configuring the Functional Policy

Functional scheduling is a nonfeedback scheme for determining a job's importance. Functional scheduling associates a job with the submitting user, project, or department. Functional scheduling is sometimes called priority scheduling. The functional policy setup ensures that a defined share is guaranteed to each user, project, job, or department at any time. Jobs of users, projects, or departments that have used fewer resources than anticipated are preferred when the system dispatches jobs to idle resources.

At the same time, full resource usage is guaranteed, because unused share proportions are distributed among those users, projects, departments, and jobs that need the resources. Past resource consumption is not taken into account.

Functional policy entitlement to system resources is combined with other entitlements in determining a job's net entitlement. For example, functional policy entitlement might be combined with override policy entitlement.

The total number of tickets that are allotted to the functional policy determines the weight of functional scheduling among the scheduling policies. During installation, the administrator divides the total number of functional tickets among the functional categories of user, department, project, and job.

Functional Shares

Functional shares are assigned to every member of each functional category: user, department, project, and job. These shares indicate the proportion of the tickets for a category to which each job associated with a member of the category is entitled. For example, user davidson has 200 shares, and user donlee has 100. A job submitted by davidson is entitled to twice as many user-functional-tickets as a job submitted by donlee.

The functional tickets that are allotted to each category are shared among all the jobs that are associated with a particular category.

Configuring the Functional Share Policy With QMON

At the bottom of the QMON Policy Configuration dialog box, click Functional Policy. The Functional Policy dialog box appears.

Function Category List

Select the functional category for which you are defining functional shares: user, project, department, or job.

Functional Shares Table

The table under Functional Shares is scrollable. The table displays the following information:

• A list of the members of the category currently selected from the Function Category list.
• The number of functional shares for each member of the category. Shares are used as a convenient indication of the relative importance of each member of the functional category. You can edit this field.
• The percentage of the functional share allocation for this category of functional ticket that this number of functional shares represents. This field is a feedback device and is not editable.

QMON periodically updates the information displayed in the Functional Policy dialog box. Click Refresh to force the display to refresh immediately.

To save all node changes that you make, click Apply. To close the dialog box without saving changes, click Done.

Changing Functional Configurations

Click the jagged arrow above the Functional Shares table to open a configuration dialog box.

• For User functional shares, the User Configuration dialog box appears. Use the User tab to switch to the appropriate mode for changing the configuration of Grid Engine users. See Configuring User Objects With QMON.
• For Department functional shares, the User Configuration dialog box appears. Use the Userset tab to switch to the appropriate mode for changing the configuration of departments that are represented as usersets. See Defining Usersets As Projects and Departments for further information.
• For Project functional shares, the Project Configuration dialog box appears. See Defining Projects With QMON.
• For Job functional shares, the Job Control dialog box appears. See Monitoring and Controlling Jobs With QMON.

Ratio Between Sorts of Functional Tickets

To display the Ratio Between Sorts Of Functional Tickets, click the arrow at the right of the Functional Shares table.

User [%], Department [%], Project [%], and Job [%] always add up to 100%.

When you change any of the sliders, all other unlocked sliders change to compensate for the change.

When a lock is open, the slider that it guards can change freely. The slider can change either because it is moved or because the moving of another slider causes this slider to change. When a lock is closed, the slider that it guards cannot change. If four locks are closed and one lock is open, no sliders can change.

• User slider – Indicates the percentage of the total functional tickets to allocate to the users category
• Departments slider – Indicates the percentage of the total functional tickets to allocate to the departments category
• Project slider – Indicates the percentage of the total functional tickets to allocate to the projects category
• Job slider – Indicates the percentage of the total functional tickets to allocate to the jobs category

Creating User-Based, Project-Based, and Department-Based Functional Scheduling

Use this setup to create a certain share assignment of all the resources in the cluster to different users, projects, or departments. First-come, first-served scheduling is used among jobs of the same user, project, or department.

1. In the Scheduler Configuration dialog box, select the Share Functional Tickets check box.
   See Sharing Functional Ticket Shares, and the sched_conf(5) man page.
   
   
2. Specify the number of functional tickets (for example, 1000000) in the scheduler configuration.
   See Configuring Policy-Based Resource Management With QMON, and the sched_conf(5) man page.
   
   
3. Add scheduling-relevant items:
   • Add one user for each scheduling-relevant user. See Configuring User Objects With QMON, and the user(5) man page.
   • Add one project for each scheduling-relevant project. See Defining Projects With QMON, and the project(5) man page.
   • Add each scheduling-relevant department.
     
4. Assign functional shares to each user, project, or department.
   See Configuring User Access Lists With QMON, and the access_list(5) man page.
   Assign the shares as a percentage of the whole. Examples follow:
   • For users:
     • UserA (10)
     • UserB (20)
     • UserC (20)
     • UserD (20)
   • For projects:
     • ProjectA (55)
     • ProjectB (45)
   • For departments:
     • DepartmentA (90)
     • DepartmentB (5)
     • DepartmentC (5)
       

Configuring the Override Policy

Override scheduling enables a Grid Engine system manager or operator to dynamically adjust the relative importance of one job or of all jobs that are associated with a user, a department, or a project. This adjustment adds tickets to the specified job, user, department, or project. By adding override tickets, override scheduling increases the total number of tickets that a user, department, project, or job has. As a result, the overall share of resources is increased.

The addition of override tickets also increases the total number of tickets in the system. These additional tickets deflate the value of every job's tickets.

You can use override tickets for the following two purposes:

• To temporarily override the share-based policy or the functional policy without having to change the configuration of these policies.
• To establish resource entitlement levels with an associated fixed amount of tickets. The establishment of entitlement levels is appropriate for scenarios like high, medium, or low priority classes.

Override tickets that are assigned directly to a job go away when the job finishes. All other tickets are inflated back to their original value. Override tickets that are assigned to users, departments, projects, and jobs remain in effect until the administrator explicitly removes the tickets.

The Policy Configuration dialog box displays the current number of override tickets that are active in the system.

Note Override entries remain in the Override dialog box. These entries can influence subsequent work if they are not explicitly deleted by the administrator when they are no longer needed.

Configuring the Override Policy With QMON

At the bottom of the Policy Configuration dialog box, click Override Policy. The Override Policy dialog box appears.

Override Category List

Select the category for which you are defining override tickets: user, project, department, or job.

Override Table

The override table is scrollable. It displays the following information:

• A list of the members of the category for which you are defining tickets. The categories are user, project, department, and job.
• The number of override tickets for each member of the category. This field is editable.

QMON periodically updates the information that is displayed in the Override Policy dialog box. Click Refresh to force the display to refresh immediately.

To save all override changes that you make, click Apply. To close the dialog box without saving changes, click Done.

Changing Override Configurations

Click the jagged arrow above the override table to open a configuration dialog box.

• For User override tickets, the User Configuration dialog box appears. Use the User tab to switch to the appropriate mode for changing the configuration of Grid Engine users. See Configuring User Objects With QMON for more details.
• For Department override tickets, the User Configuration dialog box appears. Use the Userset tab to switch to the appropriate mode for changing the configuration of departments that are represented as usersets. See Defining Usersets As Projects and Departments for more details.
• For Project override tickets, the Project Configuration dialog box appears. See Defining Projects With QMON for more details.
• For Job override tickets, the Job Control dialog box appears. See Monitoring and Controlling Jobs With QMON.
  

Configuring Policies From the Command Line

Note You must use the QMON graphical interface to perform certain policy configuration tasks, such as configuring the share tree policy in detail. However, you can use the command line to perform some simple functions.

Configuring the Share-Based Policy From the Command Line

Note Use QMON to configure the share tree policy, because a hierarchical tree is well-suited for graphical display and for editing. However, if you need to integrate share tree modifications in shell scripts, for example, you can use the qconf command and its options.

To configure the share-based policy from the command line, use the qconf command with appropriate options.

• The qconf options -astree, -mstree, -dstree, and -sstree enable you to do the following:
  • Add a new share tree
  • Modify an existing share tree
  • Delete a share tree
  • Display the share tree configuration
    See the qconf(1) man page for details about these options. The share_tree(5) man page contains a description of the format of the share tree configuration.
    
• The -astnode, -mstnode, -dstnode, and -sstnode options do not address the entire share tree, but only a single node. The node is referenced as path through all parent nodes down the share tree, similar to a directory path. The options enable you to add, modify, delete, and display a node. The information contained in a node includes its name and the attached shares.
• The weighting of the usage parameters CPU, memory, and I/O are contained in the scheduler configuration as usage_weight. The weighting of the half-life is contained in the scheduler configuration as halftime. The compensation factor is contained in the scheduler configuration as compensation_factor. You can access the scheduler configuration from the command line by using the -msconf and the -ssconf options of qconf. See the sched_conf(5) man page for details about the format.

Configuring the Functional Share Policy From the Command Line

To configure the functional share policy from the command line, use the qconf command with the appropriate options.

• Use the qconf -muser command to configure the user category. The -muser option modifies the fshare parameter of the user entry file. See the user(5) man page for information about the user entry file.
• Use the qconf -mu command to configure the department category. The -mu option modifies the fshare parameter of the access list file. See the access_list(5) man page for information about the access list file, which is used to represent departments.
• Use the qconf -mprj command to configure the project category. The -mprj option modifies the fshare parameter of the project entry file. See the project(5) man page for information about the project entry file.
• The weighting between different categories is defined in the scheduler configuration sched_conf and can be changed using qconf -msconf. The parameters to change are weight_user, weight_department, weight_project, and weight_job. The parameter values range between 0 and 1, and the total sum of parameters must add up to 1.

To assign functional shares to jobs, use the -js job_share option with the qsub, qsh, qrsh, qlogin, and qalter commands. The -js job_share option defines or redefines the job share of the job relative to other jobs. job_share is an unsigned integer value. The default job_share value for jobs is 0.

Configuring the Override Policy From the Command Line

To configure the override policy from the command line, use the qconf command with the appropriate options.

• Use the qconf -muser command to configure the user category. The -muser option modifies the oticket parameter of the user entry file. See the user(5) man page for information about the user entry file.
• Use the qconf -mu command to configure the department category. The -mu option modifies the oticket parameter of the access list file. See the access_list(5) man page for information about the access list file, which is used to represent departments.
• Use the qconf -mprj command to configure the project category. The -mprj option modifies the oticket parameter of the project entry file. See the project(5) man page for information about the project entry file.

To change the number of override tickets for the specified job, use the qalter -ot override_tickets command.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Managing Resource Quotas

This section explains how to use the resource quotas feature of the Grid Engine software to limit resources by user, project, host, cluster queue, or parallel environment. For convenience, you can express these limits using user access lists, departments, or host groups.

This section covers the following topics:

Resource Quota Overview

To prevent users from consuming all available resources, the Grid Engine software supports complex attributes that you can configure on a global, queue or host layer. While this layered resource management approach is powerful, the approach leaves gaps that become particularly important in large installations that consist of many different custom resources, user groups, and projects. The resource quota feature closes this gap by enabling you to manage these enterprise environments to the extent that you can control which project or department must abdicate when single bottleneck resources run out.

The resource quota feature enables you to apply limits to several kinds of resources and resource consumers, to all jobs in the cluster, and to combinations of consumers. In this context, resources are any defined complex attribute known by the Sun Grid Engine configuration. For more information about complex attributes, see the complex(5) man page. Resources can be slots, arch, mem_total, num_proc, swap_total, built-in resources, or any custom-defined resource like compiler_license. Resource consumers are (per) users, (per) queues, (per) hosts, (per) projects, and (per) parallel environments.

The resource quota feature provides a way for you to limit the resources that a consumer can use at any time. This limitation provides an indirect method to prioritize users, departments, and projects. To define directly the priorities by which a user should obtain a resource, use the resource urgency and share-based policies described in Configuring the Urgency Policy and Configuring the Share-Based Policy.

To limit resources through the Grid Engine software, use the qquota and qconf commands, or the QMON graphical interface. For more information, see the qquota(1) and qconf(1) man pages.

About Resource Quota Sets

Resource quota sets enable you to specify the maximum resource consumption for any job requests. Once you define the resource quota sets, the scheduler uses them to select the next possible jobs to be run by watching that the quotas will not be exceeded. The ultimate result of setting resource quotas is that only those jobs that do not exceed their resource quotas will be scheduled and run.

A resource quota set defines a maximum resource quota for a particular job request. All of the configured rule sets apply all of the time. If multiple resource quota sets are defined, the most restrictive set applies. Every resource quota set consists of one or more resource quota rules. These rules are evaluated in order, and the first rule that matches a specific request is used. A resource quota set always results in at most one effective resource quota rule for a specific request.

A resource quota set consists of the following information:

• Name – The name of the resource quota set.
• Enabled – A boolean value that indicates whether the resource set should be considered in scheduling decisions. If enabled is true, the resource quota set is active and will be considered for scheduling decisions. The default value is false.
• Description – An optional field that contains an arbitrary string that describes the set. The default value is NONE.
• Limit rule – Every resource quota set needs at least one limit rule, which is contained in the limit field. For example, the following limit rule limits all users together to 10 slots: limit users * to slots=10. The limit rule contains the following information:
  • Name – An optional name for the rule. If used, the name must be unique within the resource quota set.
  • Filter scope – The filter scope identifies the list of resource consumers to which the quota applies. A resource consumer contains a keyword followed by a comma-separated list of consumers. Use the following keywords: users, projects, queues (cluster queues), hosts, or pes (parallel environments).
    The following example shows a resource consumer and a filter scope. The defined filter scope limits user1 and user2 to the maximum number of the configured limit independently from the host.

    users {user1, user2}
    users {user1, user2} hosts *
    

    To include an expandable list in the resource quota definition, use curly braces around the resource consumer list, as shown in the resource consumer and filter scope example above.
    To exclude one of a specific resource type from a list, use ! (the exclamation point, sometimes referred to as the "not" symbol).
    

  • Limit – An attribute-value pair that defines the actual limit for the resource. For example, virtual_free=2G. You can also combine pairs into a comma-separated list of attribute-value pairs. For example, virtual_free=2G,swap_free=1.5G.

Example – Sample Resource Quota Set

The following example resource quota set restricts user1 and user2 to 2 Gbytes of free virtual space on each host in the host group lx_hosts.

     {
        name         max_virtual_free_on_lx_hosts
        description  "resource quota for virtual_free restriction"
        enabled      true
        limit        users {user1,user2} hosts {@lx_host} to virtual_free=2g
     }

Static and Dynamic Resource Quotas

Resource quota rules always define a maximum value of a resource that can be used. In most cases, these values are static and equal for all matching filter scopes. Although you could define several different rules to apply to different scopes, you would then have several rules that are nearly identical. Instead of duplicating rules, you can instead define a dynamic limit.

A dynamic limit uses an algebraic expression to derive the rule limit value. The algebraic formula can reference a complex attribute whose value is used to calculate the resulting limit.

Example – Dynamic Limit Example

The following example illustrates the use of dynamic limits. Users are allowed to use five slots per CPU on all Linux hosts.

limit hosts {@linux_hosts} to slots=$num_proc*5

The value of num_proc is the number of processors on the host. The limit is calculated by the formula $num_proc*5, and can be different on each host. Expanding the example above, you could have the following resulting limits:

• On a host with two CPUs, users can use ten slots to run jobs.
• On a host with one CPU, users can use only five slots to run jobs.

Instead of num_proc, you could use any other complex attribute known for a host as either a load value or a consumable resource.

Managing Resource Quotas With QMON

The following task explains how to set resource quotas using the QMON graphical interface.

How to Set Resource Quotas Using QMON

1. In the QMON Main Control window, click the Resource Quota Configuration button.
   
   
2. Type the Resource Quota information in the text field.
   Use the same syntax as you would for the qconf command, as illustrated in the following screen example.
   

Monitoring Resource Quota Utilization From the Command Line

Use the qquota command to view information about the current Sun Grid Engine resource quotas. The qquota command lists each resource quota that is being used at least once or that defines a static limit. For each applicable resource quota, qquota displays the following information:

• Resource quota rule – The name of the rule set name and the name or number of the rule
• Limit – The resource name, the number of available items for that resource, and the number of used items for that resource
• Filter – The effective resource quota set filter, which results from applying the filter scope explained in About Resource Quota Sets

The qquota command includes several options that you can use to limit the information to a specific host, cluster queue, project, parallel environments, resource, or user. If you use no options, qquota displays information about resource sets that apply to the user name from which you invoke the command. For more information, see the qquota(1) man page.

Example – Sample qquota Command

The following example shows information about the resource quota sets that apply to user user1:

$ qquota -u user1
resource quota    limit                filter
--------------------------------------------------------------------------------
maxujobs/1         slots=5/20           -
max_linux/1        slots=5/5            hosts @linux
max_per_host/1     slots=1/2            users user1 hosts host2

Configuring Resource Quotas From the Command Line

Use the qconf command to add, modify, or delete resource quota sets and rules.

• To add a resource quota set by invoking a text editor, use the following command:
  

  $ qconf -arqs [name]
  

  To add a set that is already defined in a file, use qconf -Arqs filename.

• To modify information about a resource quota set by invoking an editor, use the following command:
  

  $ qconf -mrqs [name]
  

  To modify a set from information contained in a file, use qconf -Mrqs filename name.
  

  Note If you use the -mrqs or -Mrqs option without a name, the new rule set replaces all the currently configured rule sets.

• To delete a resource quota set, use the following command:

  $ qconf -drqs [name_list]
  

• To see a list of defined resource quota sets, use the following command:

  $ qconf -srqsl 
  

• To view detailed information about a defined resource quota set, use the following command:

  $ qconf -srqs [name_list] 
  

For more information about qconf, see the qconf(1) man page.

Resource Quota Command Line Examples

The following example shows how you can use the various commands for resource quotas. The rule set shown in Example – Rule Set defines the following limit:

• All users together should never take more than 20 slots.
• All users should take at most 5 slots on all Linux hosts.
• Every user is restricted to 1 slot per Linux host except MyUser who is restricted to 2 slots, and all other slots on hosts are restricted to 0.
• The host group @linux includes host1 and host2.

To configure the rule set, use one of the following forms of the qconf command:

• qconf -arqs rule-set-name for each rule set
• qconf -arqs to run all rule sets at once

After jobs are submitted for different users, the qstat command shows output similar to the example shown in Example – qstat Output.

Example – Rule Set

{
 name maxujobs
 limit users * to slots=20
}

{
 name max_linux
 limit users * hosts @linux to slots=5
}

{
 name max_per_host
 limit users MyUser hosts {@linux} to slots=2
 limit users {*} hosts {@linux} to slots=1
 limit users * hosts * to slots=0
}

Example – qstat Output

$ qstat
job-ID  prior   name       user       state submit/start at     queue       slots ja-task-ID 
---------------------------------------------------------------------------------------------
     27 0.55500 Sleeper    MyUser     r     02/21/2006 15:53:10 all.q@host1   1        
     29 0.55500 Sleeper    MyUser     r     02/21/2006 15:53:10 all.q@host1   1        
     30 0.55500 Sleeper    MyUser     r     02/21/2006 15:53:10 all.q@host2   1        
     26 0.55500 Sleeper    MyUser     r     02/21/2006 15:53:10 all.q@host2   1        
     28 0.55500 Sleeper    user1      r     02/21/2006 15:53:10 all.q@host2   1        

Example – qquota Output

$ qquota # as user MyUser
resource quota rule    limit                filter
--------------------------------------------------------------------------------
maxujobs/1         slots=5/20           -
max_linux/1        slots=5/5            hosts @linux
max_per_host/1     slots=2/2            users MyUser hosts host2
max_per_host/1     slots=2/2            users MyUser hosts host1

$ qquota -h host2 # as user MyUser
resource quota    limit                filter
--------------------------------------------------------------------------------
maxujobs/1         slots=5/20           -
max_linux/1        slots=5/5            hosts @linux
max_per_host/1     slots=2/2            users MyUser hosts host2

$ qquota -u user1
resource quota    limit                filter
--------------------------------------------------------------------------------
maxujobs/1         slots=5/20           -
max_linux/1        slots=5/5            hosts @linux
max_per_host/1     slots=1/2            users user1 hosts host2

$ qquota -u *
resource quota    limit                filter
--------------------------------------------------------------------------------
maxujobs/1         slots=5/20           -
max_linux/1        slots=5/5            hosts @linux
max_per_host/1     slots=2/2            users MyUser hosts host1
max_per_host/1     slots=2/2            users MyUser hosts host2
max_per_host/1     slots=1/2            users user1 hosts host2

Performance Considerations

Efficient Rule Sets

To provide the most efficient processing of jobs and resources in queues, put the most restrictive rule at the first position of a rule set. Following this convention helps the Sun Grid Engine scheduler to restrict the amount of suited queue instances in a particularly efficient manner, because the first rule is never shadowed by any subsequent rule in the same rule set and thus always stands for itself.

To illustrate this rule, consider an environment similar to the following:

• Four queues named Q001-Q004.
• Four managed resources named F001-F004.
• Jobs that require a specific managed resource, such as F001, are constrained to run in the associated queue, such as Q001.
• Jobs are submitted into one of five projects named P001-P005.

In such an environment, you might define a single rule set as follows:

{
      name         30_for_each_project
      description  "not more than 30 per project"
      enabled      TRUE
      limit projects {*} queues Q001 to F001=30
      limit projects {*} queues Q002 to F002=30
      limit projects {*} queues Q003 to F003=30
      limit projects {*} queues Q004 to F004=30
      limit to F001=0,F002=0,F003=0,F004=0
   } 

The single rule set limits the utilization of each managed resource to 30 for each project and constrains the jobs in eligible queues at the same time. This will work fine, but in a larger cluster with many hosts, the single rule set would become the cause of slow job dispatching.

To help the Sun Grid Engine scheduler to foreclose as many queue instances as possible during matchmaking, use four separate rule sets.

{
      name         30_for_each_project_in_Q001
      description  "not more than 30 per project of F001 in Q001"
      enabled      TRUE
      limit queues !Q001 to F001=0
      limit projects {*} queues Q001 to F001=30
   }

   {
      name         30_for_each_project_in_Q002
      description  "not more than 30 per project of F002 in Q002"
      enabled      TRUE
      limit queues !Q002 to F002=0
      limit projects {*} queues Q002 to F002=30
   }

   {
      name         30_for_each_project_in_Q003
      description  "not more than 30 per project of F003 in Q003"
      enabled      TRUE
      limit queues !Q003 to F003=0
      limit projects {*} queues Q003 to F003=30
   }

   {
      name         30_for_each_project_in_Q004
      description  "not more than 30 per project of F004 in Q004"
      enabled      TRUE
      limit queues !Q004 to F004=0
      limit projects {*} queues Q004 to F004=30
   } 

These four rule sets constrain the very same per project resource quotas as the single rule set. However, the four rule sets can be processed much more efficiently due to unsuitable queue instances being shielded first. Consolidating these shields into a single resource quota set would not be doable in this case.

Note The purpose of the sample above is not to recommend one cluster queue per resource. In fact, the opposite is true, because fewer queues finally always enable fewer, more powerful shields as shown here:

  {
      name         30_for_each_project_in_Q001
      description  "not more than 30 per project of F001/F002 in Q001"
      enabled      TRUE
      limit queues !Q001 to F001=0,F002=0
      limit projects {*} queues Q001 to F001=30,F002=30
   }

   {
      name         30_for_each_project_in_Q002
      description  "not more than 30 per project of F003/F004 in Q002"
      enabled      TRUE
      limit queues !Q002 to F003=0,F004=0
      limit projects {*} queues Q002 to F003=30,F004=30   }

In this example, the queues are consolidated from Q001-Q004 down to Q001-Q002. However, this actually increases overall cluster utilization and throughput.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Managing Advance Reservations

About Advance Reservations

An advance reservation is a reservation (possibly independent of a particular job) that a user or administrator can request and the scheduler can create. This reservation causes the associated resources to be reserved for the specified user, administrator, or job. An advance reservation might limit a particular resource capability over a defined time interval. The actual resource is likely obtained by the requestor (scheduler) from the resource owner through a negotiation process.

You might better understand the concept of an advance reservation if you think about a travel reservation system. Using the Sun Grid Engine resource reservation capability, all passengers are guaranteed to get on a plane flight in the order in which the passengers arrive at the airport. What you really want is to be able to reserve your flights in advance so that you can arrange your specific flight schedule before you arrive at the airport. Grid Engine enables you to make those arrangements in advance based on an allocation scheme that the scheduler uses.

Capabilities

An advance reservation is defined by the following:

• A start time, which is defined using the standard date-time format
• An end time, which is either defined using the standard date-time format or computed from the start time plus a duration value

Advance Reservation States

Grid Engine supports the following advance reservation states:

Using QMON for Advance Reservations

How to Create Advance Reservations Using QMON

1. On the QMON main control window, click the Advance Reservation icon.
   
   
   
2. To define the advance reservation, click the Submit button.
   The AR Definition window appears, as shown in this example.
   
   
   
3. Define the time at which the advance reservation will start.

   Tip If you do not provide a start time, the advance reservation starts as soon as you submit the definition.

   
   

4. Define the time at which the advance reservation will end or provide a duration.

   Note A duration or end time is the only required parameter for an advance reservation.

   
   

5. (Optional) Provide additional parameters as desired.
   For more information about the possible parameters, click the Help button in the QMON window or see the qrsub(1) man page.
   
   
6. When you are satisfied with the parameters for your advance reservation, click the Submit AR button.
   If you see an error similar to the following, confirm that the user who is running QMON is in the arusers access list.

How to View Advance Reservations Using QMON

1. On the QMON main control window, click the Advance Reservation icon.
   The AR Control dialog box lists any defined advance reservations that are available. To sort your advance reservations, click twice on the column heading that you would like to sort by.
   
   
   By default, the columns shown in the Advance Reservation Control view are the following:
   • ARId – The Advance Reservation Id
   • ARName – The name of the Advance Reservation
   • Owner – The owner of the Advance Reservation
   • State – The state of the Advance Reservation (see Advance Reservation States)
   • StartTime – The time the Advance Reservation becomes active
   • EndTime – The time the Advance Reservation ends
   • Duration – The duration of the Advance Reservation
     
2. To add or remove non-default attributes to the display, click the Customize button.
   The AR Customize dialog box appears.
   
   
   
3. Select the attributes that you would like to display.
   
   
4. Click Cancel, Save, or Ok to exit the AR Customize dialog box.
   • Cancel dismisses any changes.
   • Ok makes the changes active as long as QMON is running.
   • Save makes the changes available permanently.

How to Delete Advance Reservations Using QMON

1. On the QMON main control window, click the Advance Reservation icon.
   
   
2. On the AR Control window, select one or more advance reservations.
   
   
3. Click the Delete button.

Note If jobs are running in the specific advance reservation, the request to delete fails. To force the advance reservation to delete anyhow, select the Force checkbox, then click Delete again.

Configuring Advance Reservations

User Access

The ability to create an advance reservation is limited to members of the arusers list.
The arusers list is created during Grid Engine installation. Only the Grid Engine administration user and those people explicitly included in the list can create advance reservations.

How to Enable a User to Create Advance Reservations

1. To add a specific user to the access list from the command line, type a command similar to the following example:

   # qconf -au username-to-add arusers
   

   You should see a confirmation message similar to the following:

   added "username-to-add" to access list "arusers"
   

Tip You can also perform this task from the QMON graphical interface. For information, see How to Configure User Access Lists with QMON.

ARCo Queries for Advance Reservations

The Accounting and Reporting Console (ARCo) provides several queries that specifically apply to advance reservations:

• Accounting per AR
• Advanced Reservation Attributes
• Advanced Reservation Log
• Advanced Reservation Time Usage
• Advanced Reservation by User
• Number of Jobs Completed per AR

For more information about ARCo, see Accounting and Reporting Console (ARCo).

Advance Reservation Command Reference

The following Grid Engine commands enable you to manage advance reservations. In addition, many standard Grid Engine commands supply information about your Advance Reservations. For example, the qsub command now includes a switch -ar that lets you specify the advance reservation into which to submit a specific job.

qrsub

Use the qrsub command to create an advance reservation and submit it to the Sun Grid Engine queuing system.

You may define default request files (analogous to sge_request for qsub) that can contain any of the possible command line options. The file names are $SGE_ROOT/$SGE_CELL/common/sge_ar_request (global defaults file) and $HOME/.sge_ar_request (user private defaults file).

Options

Many of the options for qrsub are the same as those for qsub. For more information, see the submit(1) man page.

qrsub Examples

The following example reserves an slot in the queue all.q on host1 or host2 or host3.

qrsub -q "*@host1,*@host2,*@host3" -u $user -a 01121200 -d 1:0:0

The following example reserves 4 slots on a host with arch=sol-sparc64.

qrsub -pe alloc_pe_slots 4 -l a=sol-sparc64 -u $user -a 01121200 -d 1:0:0

qrdel

Use the qrdel command to delete an advance reservation. The qrdel command requires at least one advance reservation identifier, which can be either an AR-ID (number) or an AR name. The qrdel command deletes ARs in the order in which their identifiers are presented.

Jobs referring to a advance reservation that is tagged for deletion will also be removed. Only if all jobs referring an AR are removed from the Sun Grid Engine database, the reservation also will be removed.

Options

Example

The following example deletes the advance reservation 193.

qrdel 193

qrstat

Use the qrstat command to view the current status of the granted Sun Grid Engine advance reservations. You can get information about specific ARs or users. Without any options, qrstat displays an overview of all reservations.

Options

Option	Description
-ar ar_list	Shows advance reservation information for all ARs contained in the ar_list. The ar_list can contain advance reservation IDs, names, or patterns.
-u user_list	Displays information only for those ARs being requested by one of the users from the given list. The string $user is a placeholder for the current user name. An asterisk can be used as user name wild-card to request any users ARs be displayed.
-explain	Displays the reason for a advance reservation error state. Possible reasons are: Reserved host is in unknown state. Reserved queue instance is in error state. The output format for the error reasons is one line per reason.
-xml	Prints output in XML format to stdout.
-help	Displays command usage.

Examples

The first example shows information about all advance reservations. The second example shows detailed information about the advance reservation whose ID is 193.

% qrstat
AR-ID   name       owner        state start at            end at              duration
---------------------------------------------------------------------------------------
    192 project_xy user1        r     12/14/2006 14:47:23 12/14/2006 14:57:33 0:10:10
    193            user2        w     12/18/2006 10:00:00 12/19/2006 10:00:10 24:0:10

% qrstat -ar 193
==============================================================
id:                         193
ar_name:                    
submission_time:            Mon Nov 27 17:11:34 2006
owner:                      user1
acl_list:                   user1,user2
start_time:                 Mon Dec 18 10:00:00 2006
end_time:                   Tue Dec 19 10:00:10 2006
duration:                   24:0:10
granted_slots:              all.q@host1=2,all.q@host2=1
resource_list:              myapp=2,myapp=1
...

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Managing Parallel Environments

About Parallel Environments

A parallel environment (PE) is a software package that enables concurrent computing on parallel platforms in networked environments.

A variety of systems have evolved over the past years into viable technology for distributed and parallel processing on various hardware platforms. The following are two examples of the most common message-passing environments:

• Parallel Virtual Machine (PVM) from Oak Ridge National Laboratories
• Message Passing Interface (MPI) from the Message Passing Interface Forum

Public domain as well as hardware vendor-provided implementations exist for both tools.

All these systems show different characteristics and have separate requirements. To handle parallel jobs running on top of such systems, the Grid Engine system provides a flexible, powerful interface that satisfies various needs.

The Grid Engine system enables you to run parallel jobs through the following programs:

• Arbitrary message-passing environments such as PVM or MPI. See the PVM User's Guide and the MPI User's Guide for details.
• Shared memory parallel programs on multiple slots, either in single queues or distributed across multiple queues and across machines for distributed memory parallel jobs.

Any number of different parallel environment interfaces can be configured concurrently.

Interfaces between parallel environments and the Grid Engine system can be implemented if suitable startup and stop procedures are provided. The startup procedure and the stop procedure are described in Parallel Environment Startup Procedure and in Termination of the Parallel Environment.

How to Configure Parallel Environments With QMON

1. On the QMON Main Control window, click the Parallel Environment Configuration button.
   The Parallel Environment Configuration dialog box appears.
   
   Currently configured parallel environments are displayed under PE List.
   
   
2. To display the contents of a parallel environment, select it.
   The selected parallel environment configuration is displayed under Configuration.
   
   
3. To add a new parallel environment, click Add.
   The Add/Modify PE dialog box appears. See How to Add or Modify Parallel Environments.
   
   
4. To modify a parallel environment, select it, and then click Modify.
   The Add/Modify PE dialog box appears. See How to Add or Modify Parallel Environments.
   
   
5. To delete a parallel environment, select it, and then click Delete.

How to Add or Modify Parallel Environments

1. From the Parallel Environment Configuration window, click Add or Modify as appropriate.
   The Add/Modify PE dialog box appears. If you are modifying a parallel environment, its name is displayed in the Name field.
   
   
   
2. To add a new parallel environment, type its name in the Name field.
   
   
3. In the Slots box, enter the total number of job slots that can be occupied by all parallel environment jobs running concurrently.
   
   
4. To change user access, click the icons at the right of the User Lists or Xuser Lists.
   User Lists displays the user access lists that are allowed to access the parallel environment. Xuser Lists displays the user access lists that are not allowed to access the parallel environment.
   The Select Access Lists dialog box appears.
   
   
   See Configuring User Access Lists for more information about user access lists.
   
   
5. To define the precise invocation sequence of the parallel environment startup and stop procedures, use the Start Proc Args and Stop Proc Args fields.
   See the sections Parallel Environment Startup Procedure and in Termination of the Parallel Environment.
   If no such procedures are required for a certain parallel environment, you can leave the fields empty.
   The first argument is usually the name of the start or stop procedure itself. The remaining parameters are command-line arguments to the procedures.
   A variety of special identifiers, which begin with a $ prefix, are available to pass internal runtime information to the procedures. The sge_pe(5) man page contains a list of all available parameters.
   
   
6. To define the number of parallel processes to allocate on each machine that is used by a parallel environment, use the Allocation Rule field.
   
   • Use a positive integer to set a specific number of process for each suitable host.
   • Use the special denominator $pe_slots to cause all the processes of a job to be allocated on a single host (SMP).
   • Use the denominators $fill_up and $round_robin to cause unbalanced distributions of processes among hosts.
     For more details about these allocation rules, see the sge_pe(5) man page.
     
7. To specify the method used to assess the number of slots that pending jobs with a slot range get, use the Urgency Slots field.
   The assumed slot allocation is meaningful when determining the resource-request-based priority contribution for numeric resources. You can specify an integer value for the number of slots. Specify min to use the slot range minimum. Specify max to use the slot range maximum. Specify avg to use the average of all numbers occurring within the job's parallel environment range request.
   
   
8. To specify whether the Grid Engine system generates parallel tasks or the corresponding parallel environment creates its own process, use the Control Slaves check box.
   The Grid Engine system uses sge_execd and sge_shepherd to generate parallel tasks. Full control over slave tasks by the Grid Engine system is preferable, because the system provides the correct accounting and resource control. However, this functionality is available only for parallel environment interfaces especially customized for the Grid Engine system.
   See Tight Integration of Parallel Environments and Grid Engine Software for more details.
   
   
9. If Control Slaves is checked, consider also selecting the Job Is First Task box.
   The Job Is First Task check box is meaningful only if Control Slaves is selected. If you select Job Is First Task, the job script or one of its child processes acts as one of the parallel tasks of the parallel application. For PVM, you usually want the job script to be part of the parallel application, for example. If you clear the Job Is First Task check box, the job script initiates the parallel application but does not participate. For MPI, you usually do not want the job script to be part of the parallel application, for example, when you use mpirun.
   
   
10. Click OK to save your changes and close the dialog box.
    Click Cancel to close the dialog box without saving changes.

Example – Displaying Configured Parallel Environment Interfaces With QMON

The following example defines a parallel job to be submitted. The job requests that the parallel environment interface mpi (message passing interface) be used with from 4 to 16 processes, with 16 being preferable.

• To select a parallel environment from a list of available parallel environments, click the button at the right of the Parallel Environment field. A selection dialog box appears.
  
  
  
• You can add a range for the number of parallel tasks initiated by the job after the parallel environment name in the Parallel Environment field.

Configuring Parallel Environments From the Command Line

Type the qconf command with appropriate options:

qconf <options>

The following options are available:

• qconf -ap pe-name
  The -ap option (add parallel environment) displays an editor containing a parallel environment configuration template. The editor is either the default vi editor or the editor defined by the EDITOR environment variable. pe-name specifies the name of the parallel environment. The name is already provided in the corresponding field of the template. To configure the parallel environment, change and save the template. See the sge_pe(5) man page for a detailed description of the template entries to change.
• qconf -Ap filename
  The -Ap option (add parallel environment from file) parses the specified file and adds the new parallel environment configuration. The file must have the format of the parallel environment configuration template.
• qconf -dp pe-name
  The -dp option (delete parallel environment) deletes the specified parallel environment.
• qconf -mp pe-name
  The -mp option (modify parallel environment) displays an editor containing the specified parallel environment as a configuration template. The editor is either the default vi editor or the editor defined by the EDITOR environment variable. To modify the parallel environment, change and save the template. See the sge_pe(5) man page for a detailed description of the template entries to change.
• qconf -Mp filename
  The -Mp option (modify parallel environment from file) parses the specified file and modifies the existing parallel environment configuration. The file must have the format of the parallel environment configuration template.
• qconf -sp pe-name
  The -sp option (show parallel environment) prints the configuration of the specified parallel environment to standard output.
• qconf -spl
  The -spl option (show parallel environment list) lists the names of all currently configured parallel environments.

Note To run parallel jobs, you must also associate a queue with the PE. Use the queue_conf(5) attribute pe_list to identify the suited PEs. Then, to link the PE and queues, use either the QMON utility or the following form of the qconf command: # qconf -mq <queue_name>

Example – Configuring a Parallel Environment From the Command Line

The qsub command that corresponds to the parallel job specification in Example -- Displaying Configured Parallel Environment Interfaces With QMON is as follows:

% qsub -N Flow -p -111 -P devel -a 200012240000.00 -cwd \
 -S /bin/tcsh -o flow.out -j y -pe mpi 4-16 \
 -v SHARED_MEM=TRUE,MODEL_SIZE=LARGE \
 -ac JOB_STEP=preprocessing,PORT=1234 \
 -A FLOW -w w -r y -m s,e -q big_q\
 -M me@myhost.com,me@other.address \
 flow.sh big.data

This example shows how to use the qsub -pe command to formulate an equivalent request. The qsub(1) man page provides more details about the -pe option.

Select a suitable parallel environment interface for a parallel job, keeping the following considerations in mind:

• Parallel environment interfaces can use different message-passing systems or no message systems.
• Parallel environment interfaces can allocate processes on single or multiple hosts.
• Access to the parallel environment can be denied to certain users.
• Only a specific set of queues can be used by a parallel environment interface.
• Only a certain number of queue slots can be occupied by a parallel environment interface at any point of time.

Ask your Grid Engine administrator for the available parallel environment interfaces best suited for your types of parallel jobs.

You can specify resource requirements along with your parallel environment request. The specifying of resource requirements further reduces the set of eligible queues for the parallel environment interface to those queues that fit the requirement. See Managing Resource Quotas.

For example, assume that you run the following command:

% qsub -pe mpi 1,2,4,8 -l nastran,arch=osf nastran.par

The queues that are suitable for this job are queues that are associated with the parallel environment interface mpi by the parallel environment configuration. Suitable queues also satisfy the resource requirement specification specified by the qsub -l command.

Note The parallel environment interface facility is highly configurable. In particular, the administrator can configure the parallel environment startup and stop procedures to support site-specific needs. See the sge_pe(5) man page for details. Use the qsub -v and qsub -V commands to pass information from the user who submits the job to the startup and stop procedures. These two options export environment variables. If you are unsure, ask the administrator whether you are required to export certain environment variables.

Parallel Environment Startup Procedure

The Grid Engine software starts the parallel environment by using the exec system call to invoke a startup procedure. The name of the startup executable and the parameters passed to this executable are configurable from within the Grid Engine software.

An example for such a startup procedure for the PVM environment is contained in the distribution tree of the Grid Engine software. The startup procedure is made up of a shell script and a C program that is invoked by the shell script. The shell script uses the C program to start up PVM cleanly. All other required operations are handled by the shell script.

The shell script is located under $SGE_ROOT/pvm/startpvm.sh. The C program file is located under $SGE_ROOT/pvm/src/start_pvm.c.

Note The startup procedure could have been a single C program. The use of a shell script enables easier customization of the sample startup procedure.

The example script startpvm.sh requires the following three arguments:

• The path of a host file generated by Grid Engine software, containing the names of the hosts from which PVM is to be started
• The host on which the startpvm.sh procedure is invoked
• The path of the PVM root directory, usually contained in the PVM_ROOT environment variable

These parameters can be passed to the startup script as described in Configuring Parallel Environments With QMON. The parameters are among the parameters provided to parallel environment startup and stop scripts by the Grid Engine software during runtime. The required host file, as an example, is generated by the Grid Engine software. The name of the file can be passed to the startup procedure in the parallel environment configuration by the special parameter name $pe_hostfile. A description of all available parameters is provided in the sge_pe(5) man page.

The host file has the following format:

• Each line of the file refers to a queue on which parallel processes are to run.
• The first entry of each line specifies the host name of the queue.
• The second entry specifies the number of parallel processes to run in this queue.
• The third entry denotes the queue.
• The fourth entry denotes a processor range to use in case of a multiprocessor machine.

This file format is generated by the Grid Engine software. The file format is fixed. Parallel environments that need a different file format must translate it within the startup procedure. See the startpvm.sh file. PVM is an example of a parallel environment that needs a different file format.

When the Grid Engine software starts the parallel environment startup procedure, the startup procedure launches the parallel environment. The startup procedure should exit with a zero exit status. If the exit status of the startup procedure is not zero, Grid Engine software reports an error and does not start the parallel job.

Note You should test any startup procedures first from the command line, without using the Grid Engine system. Doing so avoids all errors that can be hard to trace if the procedure is integrated into the Grid Engine software framework.

Termination of the Parallel Environment

When a parallel job finishes or is aborted, for example, by qdel, a procedure to halt the parallel environment is called. The definition and semantics of this procedure are similar to the procedures described for the startup program. The stop procedure can also be defined in a parallel environment configuration. See, for example, Configuring Parallel Environments With QMON.

The purpose of the stop procedure is to shut down the parallel environment and to reap all associated processes.

Note If the stop procedure fails to clean up parallel environment processes, the Grid Engine system might have no information about processes that are running under parallel environment control. Therefore the stop procedure cannot clean up these processes. The Grid Engine software, of course, cleans up the processes directly associated with the job script that the system has launched.

The distribution tree of the Grid Engine software also contains an example of a stop procedure for the PVM parallel environment. This example resides under $SGE_ROOT/pvm/stoppvm.sh. It takes the following two arguments:

• The path to the host file generated by the Grid Engine system
• The name of the host on which the stop procedure is started

Similar to the startup procedure, the stop procedure is expected to return a zero exit status on success and a nonzero exit status on failure.

Note You should test any stop procedures first from the command line, without using the Grid Engine software. Doing so avoids all errors that can be hard to trace if the procedure is integrated into the Grid Engine framework.

Tight Integration of Parallel Environments and Grid Engine Software

How to Configure Parallel Environments With QMON mentions that using sge_execd and sge_shepherd to create parallel tasks offers benefits over parallel environments that create their own parallel tasks. The UNIX operating system allows reliable resource control only for the creator of a process hierarchy. Features such as correct accounting, resource limits, and process control for parallel applications, can be enforced only by the creator of all parallel tasks.

Most parallel environments do not implement these features. Therefore parallel environments do not provide a sufficient interface for the integration with a resource management system like the Grid Engine system. To overcome this problem, the Grid Engine system provides an advanced parallel environment interface for tight integration with parallel environments. This parallel environment interface transfers the responsibility for creating tasks from the parallel environment to the Grid Engine software.

The distribution of the Grid Engine system contains two examples of such a tight integration, one for the PVM public domain version, and one for the MPICH MPI implementation from Argonne National Laboratories. The examples are contained in the directories $SGE_ROOT/pvm and $SGE_ROOT/mpi, respectively. The directories also contain README files that describe the usage and any current restrictions. Refer to those README files for more details.

For the purpose of comparison, the $SGE_ROOT/mpi/sunhpc/loose-integration directory contains a loose integration sample with Sun HPC ClusterTools software, and the $SGE_ROOT/mpi directory contains a loosely integrated variant of the interfaces for comparison.

Note The performance of a tight integration with a parallel environment is an advanced task that can require expert knowledge of the parallel environment and the Grid Engine software parallel environment interface. You might want to contact your Sun support representative for assistance.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Managing Checkpointing Environments

About Checkpointing

Checkpointing is a facility that does the following tasks:

• Freezes the status of an running job or application
• Saves this status (the checkpoint) to disk
• Restarts the job or application from the checkpoint if the job or application has otherwise not finished, for example, due to a system shutdown

If you move a checkpoint from one host to another host, checkpointing can migrate jobs or applications in a cluster without significant loss of resources. Hence, dynamic load balancing can be provided with the help of a checkpointing facility.

The Grid Engine system supports two levels of checkpointing:

• User-level checkpointing – At this level, providing the checkpoint generation mechanism is entirely the responsibility of the user or the application. Examples of user-level checkpointing include the following:
  • The periodic writing of restart files that are encoded in the application at prominent algorithmic steps, combined with proper processing of these files when the application is restarted
  • The use of a checkpoint library that must be linked to the application and that thereby installs a checkpointing mechanism

    Note A variety of third-party applications provides an integrated checkpoint facility that is based on the writing of restart files. Checkpoint libraries are available from hardware vendors or from the public domain. Refer to the Condor project of the University of Wisconsin, for example.

• Kernel-level transparent checkpointing – This level of checkpointing must be provided by the operating system, or by enhancements to it, that can be applied to any job. No source code changes or relinking of your application need to be provided to use kernel-level checkpointing.

Kernel-level checkpointing can be applied to complete jobs, that is, the process hierarchy created by a job. By contrast, user-level checkpointing is usually restricted to single programs. Therefore the job in which such programs are embedded needs to properly handle cases where the entire job gets restarted.

Kernel-level checkpointing, as well as checkpointing based on checkpointing libraries, can consume many resources. The complete virtual address space that is in use by the job or application at the time of the checkpoint must be dumped to disk. By contrast, user-level checkpointing based on restart files can restrict the data that is written to the checkpoint on the important information only.

About Checkpointing Environments

The Grid Engine software provides a configurable attribute description for each checkpointing method used. Different attribute descriptions reflect the different checkpointing methods and the potential variety of derivatives from these methods on different operating system architectures.

This attribute description is called a checkpointing environment. Default checkpointing environments are provided with the distribution of the Grid Engine system and can be modified according to the site's needs.

New checkpointing methods can be integrated in principal. However, the integration of new methods can be a challenging task. This integration should be performed only by experienced personnel or by your Grid Engine system support team.

How to Configure Checkpointing Environments With QMON

1. On the QMON Main Control window, click the Checkpoint Configuration button.
   The Checkpointing Configuration dialog box appears.
   
   
   
2. To view previously configured checkpointing environments, select one of the checkpointing environment names listed under Checkpoint Objects.
   The corresponding configuration is displayed under Configuration.
   
   
3. To delete a configured checkpointing environment, select it, and then click Delete.
   
   
4. To add a checkpointing environment, in the Checkpointing Configuration dialog box, click Add.
   The Add/Modify Checkpoint Object dialog box appears, along with a template configuration that you can edit.
   
   
   • Fill out the template with the requested information.
   • Click OK to register your changes with sge_qmaster.
     Click Cancel to close the dialog box without saving changes.
     
5. To modify checkpointing environments, in the Checkpoint Objects list, select the name of the configured checkpointing environment you want to modify, and then click Modify.
   The Add/Modify Checkpoint Object dialog box appears, along with the current configuration of the selected checkpointing environment.
   The Add/Modify Checkpoint Object dialog box enables you to change the following information:
   • Name
   • Interface or checkpointing method
   • Checkpoint, Migration, Restart, and Clean command strings
   • Directory where checkpointing files are stored
   • Occasions when checkpoints must be initiated
   • Signal to send to job or application when a checkpoint is initiated
     See the checkpoint(5) man page for details about these parameters.
     

     Note You must define the Interface or checkpointing method to use. From the Interface list under Name, select an Interface. See the checkpoint(5) man page for details about the meaning of the different interfaces. For the checkpointing environments provided with the distribution of the Grid Engine system, change only the Name parameter and the Checkpointing Directory parameter.

     
     

6. Click OK to register your changes with sge_qmaster.
   Click Cancel to close the dialog box without saving changes.
   
   
7. When you have configured all checkpointing environments, click Done to close the Checkpointing Configuration dialog box.

Configuring Checkpointing Environments From the Command Line

To configure the checkpointing environment from the command line, type the qconf command with the appropriate options.

The following options are available:

• qconf -ackpt ckpt-name – The -ackpt option (add checkpointing environment) displays an editor containing a checkpointing environment configuration template. The editor is either the default vi editor or the editor that corresponds to the EDITOR environment variable. The parameter ckpt-name specifies the name of the checkpointing environment. The parameter is already provided in the corresponding field of the template. To configure the checkpointing environment, change and save the template. See the checkpoint(5) man page for a detailed description of the template entries to be changed.
• qconf -Ackpt filename – The -Ackpt option (add checkpointing environment from file) parses the specified file and adds the new checkpointing environment configuration. The file must have the format of the checkpointing environment template.
• qconf -dckpt ckpt-name – The -dckpt option (delete checkpointing environment) deletes the specified checkpointing environment.
• qconf -mckpt ckpt-name – The -mckpt option (modify checkpointing environment) displays an editor containing the specified checkpointing environment as a configuration template. The editor is either the default vi editor or the editor that corresponds to the EDITOR environment variable. To modify the checkpointing environment, change and save the template. See the checkpoint(5) man page for a detailed description of the template entries to be changed.
• qconf -Mckpt filename – The -Mckpt option (modify checkpointing environment from file) parses the specified file and modifies the existing checkpointing configuration. The file must have the format of the checkpointing environment template.
• qconf -sckpt ckpt-name – The -sckpt option (show checkpointing environment) prints the configuration of the specified checkpointing environment to standard output.
• qconf -sckptl – The -sckptl option (show checkpointing environment list) displays a list of the names of all checkpointing environments currently configured.

Sun Grid Engine Information Center
Administering Sun Grid Engine
Index

Configuring Complex Resource Attributes

This section describes how to configure resource attribute definitions. Resource attribute definitions are stored in an entity called the Grid Engine system complex. This section includes the following topics:

For information about load parameters and writing your own load sensors, see Load Parameters.

About Complex Resource Attributes

The complex configuration privides all pertinent information about the resource attributes users can request for jobs with the qsub -l or qalter -l commands. The complex configuration also provides information about how the Grid Engine system should interpret these resource attributes.

The complex also builds the framework for the system's consumable resources facility. The resource attributes that are defined in the complex can be attached to the global cluster, to a host, or to a queue instance. The attached attribute identifies a resource with the associated capability. During the scheduling process, the availability of resources and the job requirements are taken into account. The Grid Engine system also performs the bookkeeping and the capacity planning that is required to prevent over-subscription of consumable resources.

Typical consumable resource attributes include:

• Available free memory
• Unoccupied licenses of a software package
• Free disk space
• Available bandwidth on a network connection

Attribute definitions in the Grid Engine complex define how resource attributes should be interpreted.

The definition of a resource attribute includes the following:

• Name of the attribute
• Shortcut to reference the attribute name
• Value type of the attribute, for example, STRING, RESTRING, TIME, or any other complex(5) type
• Relational operator used by the scheduler
• Requestable flag, which determines whether users can request the attribute for a job
• Consumable flag, which identifies the attribute as a consumable resource
• Default request value that is taken into account for consumable attributes if jobs do not explicitly specify a request for the attribute
• Urgency value, which determines job priorities on a per resource basis

Although you can define complex resource attributes from the command line, it is easier to use the QMON Complex Configuration dialog box. See:

• Configuring Complex Resource Attributes With QMON
• Configuring Complex Resource Attributes From the Command Line

Configuring Complex Resource Attributes With QMON

How to Configure Complex Resource Attributes

1. In the QMON Main Control window, click the Complex Configuration button.
   The Complex Configuration dialog box appears as shown in the following figure.
   
   The Complex Configuration dialog box enables you to add, modify, or delete complex resource attributes. See the complex(5) man page for details about the meaning of the rows and columns in the table.
   
   
2. To add a new attribute, follow these steps:
   1. Make sure that no line in the Attributes table is selected.
      To deselect a highlighted attribute, hold down the Control key and click mouse button 1.
   2. In the fields above the Attributes table, type or select the values that you want.
   3. Click the Add button.

      Tip You can add a new attribute by copying an existing attribute and then modifying it. Make sure that the attribute name and its shortcut are unique.

      
      

3. To modify an attribute, follow these steps:
   1. select the attribute in the table.
      The values of the selected attribute are displayed above the Attributes table.
   2. Change the attribute values.
   3. Click the Modify button.
      
      
4. To save configuration changes to a file, click Save.
   
   
5. To load values from a file into the complex configuration, click Load, and select the name of a file from the list.
   
   
6. To delete an attribute, select the attribute in the Attributes table, and click Delete.
   
   
7. To register your new or modified complex configuration with sge_qmaster, click Commit.

Assigning Resource Attributes to Queues, Hosts, and the Global Cluster

Resource attributes can be used in the following ways:

• As queue resource attributes
• As host resource attributes
• As global resource attributes

A set of default resource attributes is already attached to each queue and host. Default resource attributes are built in to the system and cannot be deleted, nor can their type be changed.

User-defined resource attributes must first be defined in the complex before you can assign them to a queue instance, a host, or the global cluster. When you assign a resource attribute to one of these targets, you specify a value for the attribute.

The following sections describe each attribute type in detail.

• Queue Resource Attributes
• Host Resource Attributes
• Global Resource Attributes

Queue Resource Attributes