Sun Grid Engine Information Center
Index

Sun Grid Engine Overview

Sun Grid Engine provides policy-based workload management and dynamic provisioning of application workloads. The Sun Grid Engine software enables you to apply resource management strategies to distribute jobs across a grid. A Grid Engine master can manage a grid of up to ten thousand hosts, meeting the scalability needs of even the largest grids.

If you are not yet familiar with the basic concepts and functions of the Sun Grid Engine product, you should review some of the following information before you read further:

• Introduction to Grid Engine Video
• Open Grid Engine Project
• Sun Grid Engine Product Site

For more information, choose from the following topics:

• What Is Grid Computing
• Managing Workload by Managing Resources and Policies
• How the System Operates
• Grid Engine System Components
• Interacting With Sun Grid Engine

For more information about patch requirements and patch matrix for Sun Grid Engine 6.2 Update 1 packages, see Patch Requirements and Patch Matrix for Sun Grid Engine 6.2 Update 1 Packages.

To print this section, see the Printable Sun Grid Engine Overview Guide.

Sun Grid Engine Information Center
Sun Grid Engine Overview
Index

What Is Grid Computing?

A grid is a collection of computing resources that perform tasks. In its simplest form, a grid appears to users as a large system that provides a single point of access to powerful distributed resources. In its more complex form, which is explained later in this section, a grid can provide many access points to users. In all cases, users treat the grid as a single computational resource. Resource management software such as Sun Grid Engine software accepts jobs submitted by users. The software uses resource management policies to schedule jobs to be run on appropriate systems in the grid. Users can submit millions of jobs at a time without being concerned about where the jobs run.

No two grids are alike. The three key classes of grids, which scale from single systems to supercomputer-class compute farms that use thousands of processors, are as follows:

• Cluster grids are the simplest class. Cluster grids are made up of a set of computer hosts that work together. A cluster grid provides a single point of access to users in a single project or a single department.
• Campus grids enable multiple projects or departments within an organization to share computing resources. Organizations can use campus grids to handle a variety of tasks, from cyclical business processes to rendering, data mining, and more.
• Global grids are a collection of campus grids that cross organizational boundaries to create very large virtual systems. Users have access to compute power that far exceeds resources that are available within their own organization.

The following figure shows the three classes of grids. In the cluster grid, a user's job is handled by only one of the systems within the cluster. However, the user's cluster grid might be part of the more complex campus grid, and the campus grid might be part of the largest global grid. In such cases, the user's job can be handled by any member execution host that is located anywhere in the world.

Sun Grid Engine software provides the power and flexibility required for campus grids. The product is useful for existing cluster grids because it facilitates a smooth transition to creating a campus grid. The Grid Engine system effects this transition by consolidating all existing cluster grids on the campus. In addition, the Grid Engine system is a good start for an enterprise campus that is moving to the grid computing model for the first time.

The Grid Engine software orchestrates the delivery of computational power that is based on enterprise resource policies set by the organization's technical and management staff. The Grid Engine system uses these policies to examine the available computational resources within the campus grid. The system gathers these resources and then allocates and delivers resources automatically, optimizing usage across the campus grid.

To enable cooperation within the campus grid, project owners who use the grid must do the following:

• Negotiate policies
• Develop flexible policies for manual overrides for unique project requirements
• Automatically monitor and enforce the policies

The Grid Engine software can mediate among the entitlements of many departments and projects that are competing for computational resources.

Sun Grid Engine Information Center
Sun Grid Engine Overview
Index

Managing Workload by Managing Resources and Policies

The Grid Engine system is an advanced resource management tool for heterogeneous distributed computing environments. Workload management is accomplished through managing resources and administering policies.

Sites configure the system to maximize usage and throughput, while the system supports varying levels of timeliness and importance. Job deadlines are instances of timeliness. Job priority and user share are instances of importance.

The Sun Grid Engine software provides advanced resource management and policy administration for UNIX and Windows environments that are composed of multiple shared resources. The Grid Engine system is superior to standard load management tools with respect to the following major capabilities:

• Innovative dynamic scheduling and resource management that enables the Grid Engine software to enforce site-specific management polices.
• Dynamic collection of performance data to provide the scheduler with current job-level resource consumption and system load information.
• Availability of enhanced security by way of Certificate Security Protocol (CSP)-based encryption. Instead of transferring messages in clear text, the messages in this more secure system are encrypted with a secret key.
• High-level policy administration for the definition and implementation of enterprise goals such as productivity, timeliness, and level-of-service.

You can submit computationally demanding tasks to the grid for transparent distribution of the associated workload, including batch jobs, interactive jobs, and parallel jobs.

For the administrator, the software provides comprehensive tools for monitoring and controlling jobs.

The product also supports checkpointing programs. Checkpointing jobs migrate from workstation to workstation without user intervention on load demand.

Sun Grid Engine Information Center
Sun Grid Engine Overview
Index

How the System Operates

Grid Engine System Operations

The Grid Engine system does all of the following:

• Accepts jobs from the outside world. Jobs are users' requests for computer resources.
• Puts jobs in a holding area until the jobs can be run.
• Sends jobs from the holding area to an execution device.
• Manages running jobs.
• Logs the record of job execution when the jobs are finished.

Matching Resources to Requests

As an analogy, imagine a large "money-center" bank in one of the world's capital cities. In the bank's lobby are dozens of customers waiting to be served. Each customer has different requirements. One customer wants to withdraw a small amount of money from his account. Arriving just after him is another customer, who has an appointment with one of the bank's investment specialists. She wants advice before she undertakes a complicated venture. Another customer in front of the first two customers wants to apply for a large loan, as do the eight customers in front of her.

Different customers with different needs require different types of service and different levels of service from the bank. Perhaps the bank on this particular day has many employees who can handle the one customer's simple withdrawal of money from his account. But at the same time the bank has only one or two loan officers available to help the many loan applicants. On another day, the situation might be reversed.

The effect is that customers must wait for service unnecessarily. Many of the customers could receive faster service if only their needs were immediately recognized and then matched to available resources.

If the Grid Engine system were the bank manager, the service would be organized differently:

• On entering the bank lobby, customers would be asked to declare their name, their affiliations, and their service needs.
• Each customer's time of arrival would be recorded.
• Based on the information that the customers provided in the lobby, the bank might serve the following customers in the following order:
  1. Customers whose needs match suitable and immediately available resources
  2. Customers whose requirements have the highest priority
  3. Customers who were waiting in the lobby for the longest time
• In a "Grid Engine system bank," one bank employee might be able to help several customers at the same time. The Grid Engine software would try to assign new customers to the least-loaded and most-suitable bank employee.
• As bank manager, the Grid Engine system would allow the bank to define service policies. Typical service policies might be the following:
  • To provide preferential service to commercial customers because those customers generate more profit
  • To make sure a certain customer group is served well, because those customers have received bad service in the past
  • To ensure that customers with an appointment get a timely response
  • To provide preferential treatment to certain customers because those customers were identified by a bank executive as high priority customers
• These policies would be implemented, monitored, and adjusted automatically by a Grid Engine system manager. Customers that have preferential access would be served sooner. Such customers would receive more attention from employees. The Grid Engine manager would recognize if the customers do not make progress. The manager would immediately respond by adjusting service levels to comply with the bank's service policies.

Jobs and Queues

In a Grid Engine system, jobs correspond to bank customers. Jobs wait in a computer holding area instead of a lobby. Queues, which provide services for jobs, correspond to bank employees. As in the case of bank customers, the requirements of each job, such as available memory, execution speed, available software licenses, and similar needs, can be very different. Only certain queues might be able to provide the corresponding service.

To continue the analogy, the Grid Engine software arbitrates available resources and job requirements in the following way:

• A user who submits a job through the Grid Engine software declares a requirement profile for the job. In addition, the software retrieves the identity of the user. The software also retrieves the user's affiliation with projects or user groups. The time that the user submitted the job is also stored.
• The moment that a queue is available to run a new job, the Grid Engine software determines what are the suitable jobs for the queue. The software immediately dispatches the job that has either the highest priority or the longest waiting time.
• Queues allow concurrent execution of many jobs. The Grid Engine software tries to start new jobs in the least loaded and most suitable queue.

Usage Policies

The administrator of a cluster can define high-level usage policies that are customized according to the site. Four usage policies are available:

• Urgency – Using this policy, each job's priority is based on an urgency value. The urgency value is derived from the job's resource requirements, the job's deadline specification, and how long the job waits before it is run.
• Functional – Using this policy, an administrator can provide special treatment because of a user's or a job's affiliation with a certain user group, project, and so forth.
• Share-based – Under this policy, the level of service depends on an assigned share entitlement, the corresponding shares of other users and user groups, the past usage of resources by all users, and the current presence of users within the system.
• Override – This policy requires manual intervention by the cluster administrator, who modifies the automated policy implementation.

Policy management automatically controls the use of shared resources in the cluster to best achieve the goals of the administration. High priority jobs are dispatched preferentially. Such jobs receive higher CPU entitlements if the jobs compete for resources with other jobs. The Grid Engine software monitors the progress of all jobs and adjusts their relative priorities correspondingly and with respect to the goals defined in the policies.

Using Tickets to Administer Policies

The functional, share-based, and override policies are defined through a Grid Engine concept that is called tickets. You might compare tickets to shares of a public company's stock. The more shares of stock that you own, the more important you are to the company. If shareholder A owns twice as many shares as shareholder B, A also has twice the votes of B. Therefore shareholder A is twice as important to the company. Similarly, the more tickets that a job has, the more important the job is. If job A has twice the tickets of job B, job A is entitled to twice the resource usage of job B.

Jobs can retrieve tickets from the functional, share-based, and override policies. The total number of tickets, as well as the number retrieved from each ticket policy, often changes over time.

The administrator controls the number of tickets that are allocated to each ticket policy in total. Just as ticket allocation does for jobs, this allocation determines the relative importance of the ticket policies among each other. Through the ticket pool that is assigned to particular ticket policies, the Grid Engine software can run in different ways. For example, the software can run in a share-based mode only. Or the software can run in a combination of modes, for example, 90% share-based and 10% functional.

Using the Urgency Policy to Assign Job Priority

The urgency policy can be used in combination with two other job priority specifications:

• The number of tickets assigned by the functional, share-based, and override policies
• A priority value specified by the qsub -p command

A job can be assigned an urgency value, which is derived from three sources:

• The job's resource requirements
• The length of time that a job must wait before the job runs
• The time at which a job must finish running

The administrator can separately weight the importance of each of these sources to arrive at a job's overall urgency value. For more information, see Managing Policies.

The following figure shows the correlation among policies in a Grid Engine system.

Sun Grid Engine Information Center
Sun Grid Engine Overview
Index

Grid Engine System Components

Hosts

Four types of hosts are fundamental to the Grid Engine system:

• Master host – The master host is central to the overall cluster activity. The master host runs the master daemon sge_qmaster. This daemon controls all Grid Engine system scheduling and components, such as queues and jobs. The daemon maintains tables about the status of the components, user access permissions, and the like. By default, the master host is also an administration host and a submit host.
• Execution hosts – Execution hosts are systems that have permission to execute jobs. Therefore, queue instances are attached to the execution hosts. Execution hosts run the execution daemon sge_execd.
• Administration hosts – Administration hosts are hosts that have permission to carry out any kind of administrative activity for the Grid Engine system.
• Submit hosts – Submit hosts enable users to submit and control batch jobs only. In particular, a user who is logged in to a submit host can submit jobs with the qsub command, can monitor the job status with the qstat command, and can use the Grid Engine system OSF/1 Motif graphical user interface QMON, which is described in QMON, the Grid Engine System's Graphical User Interface.

Note A system can act as more than one type of host.

Daemons

Two daemons provide the functionality of the Grid Engine system.

sge_qmaster - The Master Daemon

The center of the cluster's management and scheduling activities, sge_qmaster maintains tables about hosts, queues, jobs, system load, and user permissions. sge_qmaster also performs scheduling functions and requests actions from sge_execd on the appropriate execution hosts. sge_qmaster makes the following scheduling decisions:

• Which jobs are dispatched to which queues
• How to reorder and reprioritize jobs to maintain share, priority, or deadline

sge_execd - The Execution Daemon

The execution daemon is responsible for the queue instances on its host and for the running of jobs in these queue instances. Periodically, the execution daemon forwards information such as job status or load on its host to sge_qmaster.

Queues

A queue is a container for a class of jobs that are allowed to run on one or more hosts concurrently. A queue determines certain job attributes, for example, whether the job can be migrated. Throughout its lifetime, a running job is associated with its queue. Association with a queue affects some of the things that can happen to a job. For example, if a queue is suspended, all jobs associated with that queue are also suspended.

Jobs do not need to be submitted directly to a queue. If you submit a job to a specified queue, the job is bound to this queue. As a result, the Grid Engine system daemons are unable to select a better-suited device or a device that has a lighter load.

You only need to specify the requirement profile of the job. A profile might include requirements such as memory, operating system, available software, and so forth. The Grid Engine software automatically dispatches the job to a suitable queue and a suitable host with a light execution load.

A queue can reside on a single host, or a queue can extend across multiple hosts. For this reason, Grid Engine system queues are also referred to as cluster queues. Cluster queues enable users and administrators to work with a cluster of execution hosts by means of a single queue configuration. Each host that is attached to a cluster queue receives its own queue instance from the cluster queue.

Client Commands

The command-line user interface is a set of ancillary programs (commands) that enable you to do the following tasks:

• Manage queues
• Submit and delete jobs
• Check job status
• Suspend or enable queues and jobs

For a complete list of ancillary programs, see Client Commands. To view detailed information about each command, see the Grid Engine man pages, which are available in your $SGE_ROOT/man directory or on the Open Grid Engine site.

Sun Grid Engine Information Center
Sun Grid Engine Overview
Index

Interacting With the Sun Grid Engine Software

QMON, the Grid Engine System's Graphical User Interface

You can use QMON, the graphical user interface (GUI) tool, to accomplish most Grid Engine system tasks. These tasks include submitting jobs, controlling jobs, and gathering important information. The QMON Main Control window is often the starting point for user and administrator functions. Each icon on the Main Control window is a GUI button that you click to start a variety of tasks. To see a button's name, rest the pointer over the button. The button name describes the button function.

Launching the QMON Main Control Window

To launch the QMON Main Control window, from the command line, type qmon.

After a message window is displayed, the QMON Main Control window appears.

Figure – QMON Main Control Window, Defined

The following figure shows the QMON Main Control window along with descriptions of each icon.

Customizing QMON

A specifically designed resource file largely defines the QMON look and feel. Reasonable defaults are compiled in $SGE_ROOT/qmon/Qmon, which also includes a sample resource file. Refer to the comment lines in the sample Qmon file for detailed information on the possible customizations.

The cluster administrator can do any of the following:

• Install site-specific defaults in standard locations such as /usr/lib/X11/app-defaults/Qmon
• Include QMON-specific resource definitions in the standard .Xdefaults or .Xresources files
• Put a site-specific Qmon file in a location referenced by standard search paths such as XAPPLRESDIR

Ask your administrator if any of these cases are relevant in your environment.

In addition, users can configure the following personal preferences:

• Users can modify the Qmon file.
• The Qmon file can be moved to the home directory or to another location pointed to by the private XAPPLRESDIR search path.
• Users can include the necessary resource definitions in their private .Xdefaults or .Xresources files.
  A private Qmon resource file can also be installed using the xrdb command. The xrdb command can be used during operation. xrdb can also be used at startup of the X11 environment, for example, in a .xinitrc resource file.

You can also use the Job Customize and Queue Customize dialog boxes to customize QMON. These dialog boxes are shown in Customizing the Job Control Display and in Filtering Cluster Queues and Queue Instances. In both dialog boxes, users can use the Save button to store the filtering and display definitions to the .qmon_preferences file in their home directories. When QMON is restarted, this file is read, and QMON reactivates the previously defined behavior.

Client Commands

The command-line user interface is a set of ancillary programs (commands) that enable you to do the following tasks:

• Manage queues
• Submit and delete jobs
• Check job status
• Suspend or enable queues and jobs

The Grid Engine system provides the following set of ancillary programs:

• qacct – Extracts arbitrary accounting information from the cluster log file. For more information, see Generating Accounting Statistics.
• qalter – Changes the attributes of submitted but pending jobs.
• qconf – Provides the user interface for cluster configuration and queue configuration. For more information, see Using Files and Scripts for Administrative Tasks.
• qdel – Enables a user to delete one or more jobs. A manager or operator can delete jobs belonging to any user, while regular users can only delete their own jobs. For more information, see How to Control Jobs With qdel and qmod.
• qhold – Holds back submitted jobs from execution.
• qhost – Displays status information about execution hosts.
• qlogin – Initiates a login session with automatic selection of a low-loaded, suitable host.
• qmake – A replacement for the standard UNIX make facility. qmake extends make by its ability to distribute independent make steps across a cluster of suitable machines. For more information, see Parallel Makefile Processing With qmake.
• qmod – Enables the owner to suspend or enable a queue. All currently active processes that are associated with this queue are also signaled. For more information, see How to Control Queues With qmod and How to Control Jobs With qdel and qmod.
• qmon – Provides an X Windows Motif command interface and monitoring facility.
• qping – Checks application status of Sun Grid Engine daemons.
• qquota – Shows current usage of Sun Grid Engine resource quotas. For more information, see Monitoring Resource Quota Utilization From the Command Line.
• qrdel – Deletes Sun Grid Engine advance reservations. For more information, see qrdel.
• qresub – Creates new jobs by copying jobs that are running or pending.
• qrls – Releases jobs from holds that were previously assigned to them, for example, through qhold.
• qrsh – Can be used for various purposes, such as the following:
  • To provide remote execution of interactive applications through the Grid Engine system. qrsh is comparable to the standard UNIX facility rsh. For more information, see Remote Execution With qrsh.
  • To allow for the submission of batch jobs that, upon execution, support terminal I/O and terminal control. Terminal I/O includes standard output, standard error, and standard input.
  • To provide a submission client that remains active until the batch job finishes.
  • To allow for the Grid Engine software-controlled remote execution of the tasks of parallel jobs.
• qrstat – Shows the status of Sun Grid Engine advance reservations. For more information, see qrstat.
• qrsub – Submits an advance reservation to Sun Grid Engine. For more information, see qrsub.
• qselect – Prints a list of queue names corresponding to specified selection criteria. The output of qselect is usually sent to other Grid Engine system commands to apply actions on a selected set of queues.
• qsh – Opens an interactive shell in an xterm on a lightly loaded host. Any kind of interactive jobs can be run in this shell. For more information, see How to Submit Interactive Jobs With qsh From the Command Line.
• qstat – Provides a status listing of all jobs and queues associated with the cluster. For more information, see How to Monitor Jobs With qstat.
• qsub – The user interface for submitting batch jobs to the Grid Engine system.
• qtcsh – A fully compatible replacement for the widely known and used UNIX C shell (csh) derivative, tcsh. qtcsh provides a command shell with the extension of transparently distributing execution of designated applications to suitable and lightly loaded hosts through Grid Engine software. For more information see, Transparent Job Distribution With qtcsh.

Distributed Resource Management Application API

You can automate Sun Grid Engine functions by writing scripts that run Sun Grid Engine commands and parse the results. However, for more consistent and efficient results, you can use the Distributed Resource Management Application API (DRMAA). For more information about the DRMAA concept and how to use it with the C and Java ^TM languages, see Automating Grid Engine Functions Through the Distributed Resource Management Application API.

Community Information Members of the Open Grid Engine community might develop bindings for other languages. For information on these community-supported efforts, see http://gridengine.sunsource.net/documentation.html.