<< Previous: Installing Sun Grid Engine

Next: Loading the Distribution Files on a Workstation >>

Sun Grid Engine Information Center
Installing Sun Grid Engine
Index

Planning the Installation

Whether you have installed previous versions of the Sun Grid Engine software or this is your first time, you must do some planning before you extract and install the software. This section describes the decisions that you must make, and, wherever possible, gives you criteria on which you can base your decisions. This section consists of the following topics:

Decisions That You Must Make

You must make several decisions before you can plan the installation:

• Decide whether your system of networked computer hosts that run Sun Grid Engine 6.2 software is to be a single cluster or a collection of sub-clusters, called cells. Cells enable you to install separate instances of the Grid Engine software but share the binary files across those instances.
• Select the machines that are to be Grid Engine system hosts. Determine the host type of each machine: master host, shadow master host, administration host, submit host, execution host, or a combination.
• Ensure that all users of the Grid Engine system have the same user names on all submit and execution hosts.

  Note Hosts running Windows as an operating system cannot have the master host or shadow master host host types.

• Decide how to order Grid Engine software directories. For example, you could organize directories as a complete tree on each workstation, or you could cross-mount directories, or you could set up a partial directory tree on some workstations. You must also decide where to locate each Grid Engine software installation directory, sge-root.
• Decide on the site's queue structure.
• Determine whether to define network services as an NIS file or as local to each workstation in /etc/services.
• Use the information in the following section to gather the information that you need to complete the installation worksheet.

Gather the Necessary Information

Before you install the Grid Engine software, you must plan how to achieve the results that fit your environment. This section helps you make the decisions that affect the rest of the procedure. Write down your installation plan in a table similar to the following example. You can view the worksheet alone (for printing).

If you are going to install Grid Engine 6.2 on a Windows system, acquire and install Microsoft Services For UNIX. See Microsoft Services For UNIX for more information.

If you are going to install Grid Engine 6.2 on a Windows system, create the required Certificate Security Protocol (CSP) certificates before installing Grid Engine. See How to Install a CSP-Secured System for information about CSP certificates.

Check Other Grid Engine Installation Issues for applicability.

Disk Space Requirements

The Grid Engine software directory tree has the following fixed disk space requirements:

• 50 Mbytes for the installation files without any binaries
• Between 60 and 100 Mbytes for each set of binaries

The ideal disk space for Grid Engine system spool directories is as follows:

• 50-200 Mbytes for the master host spool directories
• 50-200 Mbytes for the Berkeley DB spool directories

The spool directories of the master host and of the execution hosts are configurable and need not reside under the default location, sge-root.

Note You must satisfy several Windows platform-specific prerequisites before you can install Grid Engine on hosts that are running the Windows operating system. You might need to install additional software on your computer which might require additional disk space. See Microsoft Services For UNIX.

$SGE_ROOT Directory

You must create a directory into which to load the contents of the distribution media. This directory is called the root directory, or $SGE_ROOT. When the Grid Engine system is running, this directory stores the current cluster configuration and all other data that must be spooled to disk.

Note For efficient spooling, place the spooling directories somewhere other than within $SGE_ROOT.

Use a valid path name for the directory that is network-accessible on all hosts. For example, if the file system is mounted using automounter, set $SGE_ROOT to /usr/SGE6, not to /tmp_mnt/usr/SGE6.

Note Throughout this information space, the $SGE_ROOT environment variable is used to refer to the directory into which the Sun Grid Engine software is installed.

The $SGE_ROOT directory is the top level of the Grid Engine software directory tree. On startup, each Grid Engine software component in a cell needs read access to the $SGE_ROOT/$SGE_CELL/common directory. When Grid Engine software is installed as a single cluster, the value of $SGE_CELL is default.

For ease of installation and administration, this directory should be readable on all hosts on which you intend to run the Grid Engine software installation procedure. For example, you can select a directory that is available across a network file system, such as NFS. If you choose to select file systems that are local to the hosts, you must copy the installation directory to each host before you start the installation procedure for the particular machine. See File Access Permissions for a description of required permissions.

Directory Organization

When determining the directory organization, you must decide the following:

• The directory organization, for example, whether you will install a complete software tree on each workstation, cross-mounted directories, or a partial directory tree on some workstations.
• Where to locate each $SGE_ROOT root directory.

Note Because changing the installation directory or the spool directories requires a new installation of the system, use extra care to select a suitable installation directory. Note that all important information from a previous installation can be preserved.

By default, the installation procedure installs the Grid Engine software, man pages, spool areas, and the configuration files in a directory hierarchy under the installation directory as shown in the following figure. If you accept this default behavior, you should install or select a directory with the access permissions that are described in File Access Permissions.

Figure – Sample Directory Hierarchy

You can choose to put the spool areas in other locations during the primary installation. See Configuring Queues for more detailed instructions.

Cells

You can set up the Grid Engine system as a single cluster or as a collection of loosely coupled clusters called cells. The $SGE_CELL environment variable indicates the cluster being referenced. When the Grid Engine system is installed as a single cluster, $SGE_CELL is not set, and the value default is assumed for the cell value.

Cluster Name

The $SGE_CLUSTER_NAME environment variable supports unique naming of the cluster. Unlike the $SGE_CELL variable, there are restrictions on $SGE_CLUSTER_NAME. If you decide to use Grid Engine SMF services on Solaris 10 or later hosts, you must select a new $SGE_CLUSTER_NAME. This name becomes part of the name of the Sun Grid Engine SMF services. The $SGE_CLUSTER_NAME is also used to distinguish multiple rc files for different clusters.

note If your $SGE_CELL name already reflects the desired cluster name and also satisfies $SGE_CLUSTER_NAME restrictions, set the cluster name to the $SGE_CELL value. Otherwise, the proposed default value is pSGE_QMASTER_PORT, which uniquely identifies the running cluster by the port on which its qmaster daemon is running. See SMF administration for more information.

User Names

For the Grid Engine system to verify that users submitting jobs have permission to submit them on the desired execution hosts, users' names must be identical on the submit and execution hosts. You might therefore have to change user names on some machines, because Grid Engine user names map directly to system user accounts.

Note User names on the master host are not relevant for permission checking. These user names do not have to match or even exist.

Installation Accounts

You can install the Grid Engine software either as the root user or as an unprivileged user, for example, your own user account. However, if you install the software when you are logged in as an unprivileged user, the installation allows only that user to run Grid Engine jobs. Access is denied to all other accounts. Installing the software when you are logged in as root resolves this restriction. However, root permission is required for the complete installation procedure. Also, if you install as an unprivileged user, you are not allowed to use the qrsh, qtcsh, or qmake commands, nor can you run tightly integrated parallel jobs.

Note To use SMF on Solaris 10 or later hosts and control the Grid Engine software services as an unprivileged user, perform the following additional steps as root user (or user with appropriate permissions): For a local user: Create the solaris.smf.manage.sge authorization echo "solaris.smf.manage.sge:::Manage Grid Engine Services::" >> /etc/security/auth_attr Create the new role sge_smf: roleadd -c "Grid Engine SMF Administrator" -g <group> -d <home_dir> -u <UID> \ -s <profile_shell> -A "solaris.smf.manage.sge" "sge_smf" Assign the just-created role sge_smf to the user: usermod -R "sge_smf" <normal_user> For a distributed name service, such as NIS, NIS+, or LDAP: Create the solaris.smf.manage.sge authorization on your name server echo "solaris.smf.manage.sge:::Manage Grid Engine Services::" >> /etc/security/auth_attr Create the Grid Engine SMF Management profile on your name server echo "Grid Engine SMF Management:::Can manage Grid Engine SMF services:auths=solaris.smf.manage.sge;" >> /etc/security/prof_attr Create the new role sge_smf and assign it to the user: /usr/sadm/bin/smrole add -D <domain_name> - -n "sge_smf" -a <normal_user> \ -d <home_dir> -c "Grid Engine SMF Administrator" -p "Grid Engine SMF Management" After this is done a non-root user <normal_user> will be able to manage Grid Engine SMF services via svcadm command or the startup scripts.

File Access Permissions

If you install the software logged in as root, you might have a problem configuring root read/write access for all hosts on a shared file system. Therefore, you might have problems putting the $SGE_ROOT files onto a network-wide file system.

You can force Grid Engine software to run all Grid Engine system components through a non-root administrative user account, for example sgeadmin. With this setup, this particular user needs only read/write access to the shared $SGE_ROOT file system.

The installation procedure asks whether files should be created and owned by an administrative user account. If you answer "Yes" and provide a valid user name, files are created by this user. Otherwise, the user name under which you run the installation procedure is used. Create an administrative user, and answer "Yes" to this question.

Make sure in all cases that the account used for file handling on all hosts has read/write access to the $SGE_ROOT directory. Also, the installation procedure assumes that the host from which you access the Grid Engine software distribution media can write to the $SGE_ROOT directory.

Note The name of the root user on Windows hosts depends on the system language of the Windows operating system. You can even change the name of the root user. The default name for many languages is the name Administrator. If your Windows host is a member of a Windows domain, only the local Administrator is the root user. Neither the members of the Administrators group, nor the domain Administrator, nor a member of the Domain Admins group are the root user. See User Management for Sun Grid Engine on Windows Hosts for more information about users on Windows hosts.

Network Services

Determine whether your site's network services are defined in an NIS database or in an /etc/services file that is local to each workstation. If your site uses NIS, determine the host name of your NIS server so that you can add entries to the NIS services map.

The Grid Engine system services are sge_execd and sge_qmaster. To add the services to your NIS map, choose reserved, unused port numbers. The following examples show sge_qmaster and sge_execd entries.

sge_qmaster 6444/tcp
sge_execd 6445/tcp

Master Host

The master host controls the Grid Engine system. This host runs the master daemon sge_qmaster.

The master host must comply with the following requirements:

• The host must be a stable platform.
• The host must not be excessively busy with other processing.
• At least 60 to 120 Mbytes of unused main memory must be available to run the Grid Engine system daemons. For very large clusters that include many hundreds or thousands of hosts and tens of thousands of jobs in the system at any time, 1 GByte or more of unused main memory might be required and 2 CPUs might be beneficial.
• The master host must be installed before shadow master execution, administration, or submit hosts.
• (Optional) The Grid Engine software directory, $SGE_ROOT, should be installed locally to cut down on network traffic.

  Note Windows hosts cannot act as master hosts.

Shadow Master Hosts

These hosts back up the functionality of sge_qmaster in case the master host or the master daemon fails. To be a shadow master host, a machine must have the following characteristics:

• It must run sge_shadowd.
• It must share sge_qmaster status, job information, and queue configuration information that is logged to disk. In particular, the shadow master hosts need read/write root or administration user access to the sge_qmaster spool directory and to the $SGE_ROOT/$SGE_CELL/common directory.
• The $SGE_ROOT/$SGE_CELL/common/shadow_masters file must contain a line defining the host as a shadow master host.

  Note If no cell name is specified during installation, the value of $SGE_CELL is default.

The shadow master host facility is activated for a host as soon as these conditions are met. You do not need to restart the Grid Engine system daemons to make a host into a shadow master host.

Note Windows hosts cannot act as shadow master hosts.

Spool Directories under the Root Directory

During the installation of the master host, you must specify the location of a spooling directory. This directory is used to spool jobs from execution hosts that do not have a local spooling directory.

Note If you are using a Windows execution host, you must use the local spooling directory.

• On the master host, spool directories are maintained under qmaster-spool-dir. The location of qmaster-spool-dir is defined during the master host installation process. The default value of qmaster-spool-dir is $SGE_ROOT/$SGE_CELL/spool/qmaster.
• On each execution host, a spool directory called execd-spool-dir is defined during the execution host installation processes. The default value of execd-spool-dir is $SGE_ROOT/$SGE_CELL/spool/execution-host-name. You will get better performance from execution hosts with local spooling directories than from execution hosts that have NFS mounted the master host's spooling directory.

  Note If no cell name is specified during installation, the value of $SGE_CELL is default.

You do not need to export these directories to other machines. However, exporting the entire $SGE_ROOT tree and making it write-accessible for the master host and all executable hosts makes administration easier.

Note If you use a Lustre fileshare as the spool directory, you should disable file striping for these directories. Refer to the Lustre Operations Manual for information on how to disable file striping.

Choosing Between Classic Spooling and Database Spooling

During the installation, you are given the option to choose between classic spooling and Berkeley DB spooling. If you choose Berkeley DB spooling, you are then given the option to spool to a local directory or to a separate host, known as a Berkeley DB spooling server.

Using a Berkeley DB spooling server might provide better performance than classic spooling. Part of this performance increase is because the master host can make non-blocking writes to the database, but has to make blocking writes to the text file used by classic spooling. Also consider file format and data integrity. Writing to the Berkeley DB provides a greater level of data integrity than writing to a text file. However, a text file stores data in a format that you can read and edit. Normally, you do not need to read these files, but the spooling directory contains the messages from the system daemons, which can be useful for debugging.

Database Server and Spooling Host

The master host can store its configuration and state to a Berkeley DB spooling database. The spooling database can be installed on the master server or on a separate host. When the Berkeley DB spools into a local directory on the master host, the performance is better. If you want to set up a shadow master host, you need to use a separate Berkeley DB spooling server (host). In this case, you have to choose a host with a configured RPC service. The master host connects through RPC to the Berkeley DB.

Note This configuration does not provide a High-Availability (HA) solution. For example, scripts of pending jobs are not spooled through BDB spool server and thus are not available for a shadow master.

With the introduction of NFS4 software available with the Solaris ^TM 10 operating system, you can use Berkeley DB spooling on a network file system. You could not use Berkeley DB spooling on previous NFS versions. This circumstance allows a shadow host installation spooled on Berkeley DB without setting up an additional Berkeley DB Spooling Server.

Caution Although using a shadow master host is more reliable, using a separate Berkeley DB spooling host results in a potential security hole. RPC communication as used by the Berkeley DB can be easily compromised. Only use this alternative if your site is secure and if users can be trusted to access the Berkeley DB spooling host by means of TCP/IP communication.

If you choose to use Berkeley DB spooling without a shadow master, you do not need to set up a separate spooling server. Likewise, if you choose not to use Berkeley DB spooling, you can set up a shadow master host without setting up a separate spooling server.

Once you determine whether you need a separate spooling server, you will also need to determine the location for the spooling directory. The spooling directory must be local to the spooling server. A default value for the location of the spooling directory is recommended during installation, but this default value is not suitable when the file server is different from the master host.

The requirements for the Berkeley DB spooling host are similar to the requirements for the master host:

• The host must be a stable platform.
• The host must not be excessively busy with other processing.
• At least 60 to 120 Mbytes of unused main memory must be available to run the Grid Engine system daemons. For very large clusters that include many hundreds or thousands of hosts and tens of thousands of jobs in the system at any time, one GByte or more of unused main memory might be required and two CPUs might be beneficial.
• (Optional) A separate spooling host must be installed before the master host.
• (Optional) The $SGE_ROOT directory should be installed locally, to cut down on network traffic.

Execution Hosts

Execution hosts run the jobs that users submit to the Grid Engine system. An execution host must first be set up as an administration host. You run an installation script on each execution host. For more information, see How to Install Execution Hosts.

Group IDs

You need to provide a range of IDs that will be assigned dynamically for jobs. The range must be big enough to provide enough numbers for the maximum number of Grid Engine jobs running at a single moment on a single host.

A group ID is assigned to each Grid Engine job to monitor the resource utilization of the job. Each job will be assigned a unique ID while it is running. For example, a range of 20000-20100 allows 100 jobs to run concurrently on a single host. You can change the group ID range for your cluster configuration at any time, but the values in the UNIX group ID range must be unused on your system.

Administration Hosts

Operators and managers of the Grid Engine system use administration hosts to perform administrative tasks such as reconfiguring queues or adding Grid Engine users.

The master host installation script automatically makes the master host an administration host. During the master host installation process, you can add other administration hosts. You can also manually add administration hosts on the master host at any time after installation.

Submit Hosts

Jobs can be submitted and controlled from submit hosts. The master host installation script automatically makes the master host a submit host.

Cluster Queues

The installation procedure creates a default cluster queue structure, which is suitable for getting acquainted with the system. The default queue can be removed after installation.

Note No matter what directory is used for the installation of the software, the administrator can change most settings that were created by the installation procedure. This change can be made while the system is running.

Consider the following when determining a queue structure:

• Whether you need cluster queues for sequential, interactive, parallel, and other job types
• Which queue instances to put on which execution hosts
• How many job slots are needed in each queue

For more detailed information on administering cluster queues, see Configuring Queues.

Scheduler Profiles

You can choose from three scheduler profiles during the installation process: normal, high, and max. You can use these predefined profiles as a starting point for Grid Engine tuning.

Using these profiles, you can optimize the scheduler for one or more of the following:

• The amount of information that is tracked about a scheduling run
• The load adjustment during a scheduling run
• Interval scheduling (the default) or immediate scheduling

You can choose from three scheduler profiles:

• normal – This profile uses load adaptation and interval scheduling, and reports all the information that the scheduler gathers during the dispatch cycle. This profile is the starting point for most grids. Use this profile if your highest priority is gathering and reporting information about a scheduling run.
• high – This profile is more appropriate for a large cluster, where throughput is more important than gathering and reporting all the information from the scheduler. This profile also uses interval scheduling. Use this profile if you want to get better performance at the cost of getting less information about your scheduling runs.
• max – This profile disables all information gathering and reporting, enables immediate scheduling, and disables load adaptation. Immediate scheduling is very useful for sites with high throughput and very short running jobs. The advantage of immediate scheduling decreases as runtime of the jobs increases. This profile can be used in clusters of any size where only throughput is important and everything else is a lower priority.

For more information on how to configure scheduling, see Administering the Scheduler.

Installation Method

Several methods are available for installing the Grid Engine software:

• Interactive
• Interactive, with increased security
• Automated, using the inst_sge script and a configuration file
• Upgrade

To decide which installation method you should use, consider the following factors.

• Do you already have the Grid Engine software installed and running?
  • If so, you will probably want to upgrade. The upgrade process is described in Upgrading From a Previous Version of Sun Grid Engine Software.
  • If not, the master host installation is only done once. The master host is typically installed interactively, as described in Installing the Sun Grid Engine Software Interactively.
• Do you need to install just a few execution hosts?
  If so, then you will probably want to install them interactively, as described in Installing the Sun Grid Engine Software Interactively.
• Do you need to install a large number of execution hosts?
  If so, then you might want to perform automated installation, using the inst_sge script and a configuration file. See Using the inst_sge Utility and a Configuration Template.
• Do you require your grid to use encryption?
  If so, you have to perform an interactive installation with increased security. See Installing the Increased Security Features.

Check the Other Installation Issues Appendix

If you are installing Grid Engine on a Linux system or on a system with IPMP, see Other Grid Engine Installation Issues for important information.

<< Previous: Installing Sun Grid Engine

Next: Loading the Distribution Files on a Workstation >>