Installing a JMX-Enabled System

Searching Sun Grid Engine 6.2

Sun Grid Engine Information Center
Installing Sun Grid Engine
Index

Installing a JMX-Enabled System

The JMX agent functionality enables access to a subset of sge_qmaster functionality via the JMX protocol. For Sun Grid Engine 6.2, the main purpose of the JMX agent is to provide an interface between the SDM Grid Engine adapter and the Sun Grid Engine system.

Additional Setup Required

The steps required to set up the JMX agent feature of Grid Engine are similar to the standard setup. You generally follow the instructions in Planning the Installation, Loading the Distribution Files on a Workstation, How to Install the Master Host, How to Install Execution Hosts and Registering Administration Hosts.
However, you have to perform a few additional tasks:

  • Generating necessary configuration files on the master host by calling the installation script with the -jmx flag and depending on the JMX specific installation settings the optional generation of certificates, keys and keystore files.
  • Optional distribution of security relevant files to the execution and submit hosts using a secure method such as ssh
  • Generating user keys, certificates and keys automatically, after master installation
  • Adding new users
  • Tweaking of JMX-specific files

How to Install a JMX Agent-Enabled System

Install the Grid Engine software as outlined in Performing an Interactive Installation, with the following exception: use the additional flag -jmx when invoking the qmaster installation scripts.

To install a JMX agent enabled system do the following:

  1. Change the master host installation procedure.
    Type the following command and respond to the prompts from the installation script. 
    # ./install_qmaster -jmx [-csp]
  2. Supply the following information to generate necessary configuration files and optionally the certificates, keys and keystores:
    • JMX agent options
      • JAVA_HOME (mandatory)
      • Additional JVM arguments (optional)
      • JMX MBean server port (mandatory)
      • JMX ssl authentication (default: true)
      • JMX ssl client authentication (default: true)
      • JMX ssl server keystore path
        (/var/sgeCA/{sge_qmaster| port$SGE_QMASTER_PORT}/$SGE_CELL/private/keystore)
      • JMX ssl server keystore password
    • Optional certificate specific options, if there is no CA available
      • Two-letter country code, for example, US for the United States
      • State
      • Location, such as a city
      • Organization
      • Organizational unit
      • CA email address
        As the installation proceeds, several JMX specific configuration files are installed:
        **jvm_threads is set to 1 instead of 0 if JMX is enabled in $SGE_ROOT/$SGE_CELL/common/bootstrap: 
        ...
jvm_threads             1
...
    • Several JMX agent specific configuration files are generated as: 
      $SGE_ROOT/$SGE_CELL/common/jmx/jaas.config
$SGE_ROOT/$SGE_CELL/common/jmx/java.policy
$SGE_ROOT/$SGE_CELL/common/jmx/jmxremote.access
$SGE_ROOT/$SGE_CELL/common/jmx/jmxremote.password
$SGE_ROOT/$SGE_CELL/common/jmx/logging.properties
$SGE_ROOT/$SGE_CELL/common/jmx/management.properties

      For a detailed description, see the comments in the files and the description below.

      Optionally the Certificate Authority is created. The directories that contain information relevant to security are as follows:

      • The publicly accessible CA and daemon certificate are stored in $SGE_ROOT/$SGE_CELL/common/sgeCA
      • The publicly accessible user certificates are stored in $SGE_ROOT/$SGE_CELL/common/sgeCA/usercerts
      • The corresponding private keys and keystore are stored in /var/sgeCA/{sge_qmaster| port$SGE_QMASTER_PORT}/$SGE_CELL/private
      • User keys, certificates and keystore are stored in /var/sgeCA/{sge_qmaster| port$SGE_QMASTER_PORT}/$SGE_CELL/userkeys/$USER
  3. The script prompts you for site information.

  4. Confirm whether the information you supplied is correct.

  5. Continue the installation.
    After the security-related setup of the master host is finished, the script prompts you to continue with the rest of the installation procedure, as in the following example: 
    SGE startup script
--------------------

Your system wide SGE startup script is installed as:

     "/scratch2/eddy/sge_sec/$SGE_CELL/common/sgemaster"

Hit Return to continue >>


  6. Proceed to the next task.
    For more information, see How to Generate Certificates and Private Keys for Users.

How to Generate Certificates, Private Keys and Keystores for Users

To use the CSP-secured system, the user must have access to a user-specific certificate, private key and keystore. Usually the steps outlined in How to Generate Certificates and Private Keys for Users are performed. After that the following procedure can be done to generate the corresponding keystore files for the users.

  1. As root on the master host run the following command: 
    # $SGE_ROOT/util/sgeCA/sge_ca -userks [-kspwf <kspwf-file>]


  2. Confirm that the creation has been successful. 
    # ls -lR /var/sgeCA/port$SGE_QMASTER_PORT/$SGE_CELL/userkeys
/var/sgeCA/port$SGE_QMASTER_PORT/$SGE_CELL/userkeys/:
total 8
drwx------   2 eddy 	staff        512 Mar 13 11:33 eddy
drwx------   2 sarah 	staff        512 Mar 13 11:33 sarah
drwx------   2 leo 	staff        512 Mar 13 11:33 leo

/var/sgeCA/port$SGE_QMASTER_PORT/$SGE_CELL/userkeys/eddy:
total 16
-rw-------   1 eddy staff       1586 Mar 13 11:32 cert.pem
-rw-------   1 eddy staff        891 Mar 13 11:32 key.pem
-rw-------   1 eddy staff       3031 Mar 13 11:33 keystore
-rw-------   1 eddy staff       1024 Mar 13 11:32 rand.seed
-rw-------   1 eddy staff        818 Mar 13 11:32 req.pem
...

Checking Certificates, Keys and Keystores

To confirm that these files contain the intended information, use the following commands:

  • To display a certificate: 
    $SGE_ROOT/util/sgeCA/sge_ca -print <whereever>/cert.pem
  • To display a key: 
    $SGE_ROOT/util/sgeCA/sge_ca -printkey <whereever>/key.pem

To display a keystore or truststore:

$JAVA_HOME/bin/keytool -list -v -keystore <whereever>/keystore

The keystore password must be entered to see all entries otherwise only the certificates are visible. For more information, see Java keytool documentation.

JMX Configuration Files

The following configuration files are installed into $SGE_ROOT/$SGE_CELL/common/jmx and are explained in detail here. Manual modification is usually not necessary and the preinstalled configurations should be sufficient.

jaas.config

Before using the JMX interface, you must run a special authentication against sge_qmaster. This process adds the correct principle that gives you the necessary role to access the JMX interfaces in read-only or read-write mode. The responsible section in the jaas.config file is named GridwareConfig or TestConfig (for testing only).
In general, the jaas.config file defines which login modules are used for which application case. The choice of the login module is defined either in a configuration file like management.properties or programmatically.
The jaas.config file contains different sections and allows the replacement of the authentication mechanism, e.g. authentication via unix pam or via LDAP (see the GridwareConfig section and TestConfig section below). The different modules can be stacked. For a general overview of Jaas, see http://java.sun.com/developer/technicalArticles/Security/jaasv2/. Here the procedure consists of two steps in the GridwareConfig section:

  • Authenticate the user (for example with keystore or Unix login or with LDAP).
  • In the JGDILogin module, add the JMXPrincipal that gives the defined role to the user. This role is used later in the jmx.access file to check if the user has read-only or read-write access.
/*
 * Default login configuration for qmaster's jmx server
 */
GridwareConfig {

    /**
     *  Accepts all clients which have a certificate which is signed with
     *  the CA certificate.
     */
    com.sun.grid.security.login.GECATrustManagerLoginModule requisite
         caTop="${com.sun.grid.jgdi.caTop}";

    /*
     *  Accepts all clients which has a valid username/password.
     *
     *  The username/password validation is done with the authuser binary (included
     *  in the Grid Engine distribution, $SGE_ROOT/utilbin/$ARCH/authuser).
     *
     *  ATTENTION: The authuser binary needs the suid bit. It does not work if grid
     *  engine is installed on a nosuid file system.
     *
     */
    com.sun.grid.security.login.UnixLoginModule requisite
        sge_root="${com.sun.grid.jgdi.sgeRoot}"
        auth_method="system";

    /*
     * Username password authentication against LDAP.
     *
     * Alternative username/password authentication if
     * com.sun.grid.security.login.UnixLoginModule is not working.
     *
     * The LDAP specific parameters have to be adjusted to the local requirements
     * For details please have a look at the LdapLoginModule javadocs.
     *
     * ATTENTION: The LdapLoginModule is only available in java 6. The
     * parameter libjvm_path must point to a java 6 jvm
     * (qconf -sconf | grep libjvm_path)
     */
    /*
    com.sun.security.auth.module.LdapLoginModule requisite
        userProvider="ldap://sun-ds/ou=people,dc=sun,dc=com"
        userFilter="(&(uid={USERNAME})(objectClass=inetOrgPerson))"
        useSSL=false;
    */

    /*
     *  The JGDILoginModule adds a JGDIPrincipal to the subject. The username of
     *  the JGDIPrincipal is the name of the first trusted principal. This name
     *  treated as username for gdi communication.
     *  For each login a new jgdi session id is created.
     *
     *  In the jmxremote.access file users who can access the system are defined
     *  Any principal matching these entries is given the corresponding role.
     *  Usually a jmxPrincipal is defined to give a user access to the system.
     *  (e.g. com.sun.grid.security.login.UserPrincipal = xyz &
     *        jmxPrincipal="controlRole" gives user xyz access under controlRole
     *  )
     */
    com.sun.grid.jgdi.security.JGDILoginModule optional
        trustedPrincipal="com.sun.grid.security.login.UserPrincipal"
        trustedPrincipal1="com.sun.security.auth.UserPrincipal"
        jmxPrincipal="controlRole";
};

/*
 *  TestConfig accepts any user. Only for testing
 */
TestConfig {

    com.sun.grid.security.login.TestLoginModule requisite;

    com.sun.grid.jgdi.security.JGDILoginModule optional
        trustedPrincipal="com.sun.grid.security.login.UserPrincipal"
        jmxPrincipal="controlRole";
};

/*
 *  Mandatory if native jgdi is used with a csp system
 *  (e.g. jgdish in csp mode)
 */
jgdi {
   com.sun.security.auth.module.KeyStoreLoginModule required
                                                    keyStoreURL="file:./keystore"
                                                    debug=false;
};

java.policy

The java.policy file that is used by the JGDIAgent restricts the possibilities of code that can be run in sge_qmaster's JVM.

Usually changes here are only necessary to change the access to a subset of the overall functionality. To tweak the policy settings to your needs it is useful to run the JMX server with security debugging enabled and to consult the generated logging files. (qconf -mconf, additional_jvm_args = -Djavax.net.debug=ssl -Djava.security.debug=access,failure)

/*
**
** with LdapLoginModule
** grant principal com.sun.security.auth.UserPrincipal "controlRole"
**
** with jmxremote.password
** grant principal javax.management.remote.JMXPrincipal "controlRole"
**
*/
grant codeBase "file:${com.sun.grid.jgdi.sgeRoot}/lib/jgdi.jar"  {
   permission java.net.SocketPermission   "*:1024-", "accept,connect";
   permission java.net.SocketPermission   "localhost:1024-", "listen,resolve";
   permission java.lang.RuntimePermission "loadLibrary.jgdi";
   permission java.lang.RuntimePermission "shutdownHooks";
   permission java.lang.RuntimePermission "setContextClassLoader";
   permission javax.security.auth.AuthPermission "createLoginContext.jgdi";
   permission javax.security.auth.AuthPermission "doAs";
   permission javax.security.auth.AuthPermission "getSubject";
   permission java.util.PropertyPermission "*", "read";
   permission java.util.logging.LoggingPermission "control";

   permission java.lang.FilePermission "${com.sun.grid.jgdi.sgeRoot}/${com.sun.grid.jgdi.sgeCell}/common/jmx/-", "read";
   permission java.io.FilePermission "${com.sun.grid.jgdi.sgeRoot}/util/-", "execute";
   permission java.io.FilePermission "${com.sun.grid.jgdi.sgeRoot}/utilbin/-", "execute";
   permission javax.management.MBeanServerPermission "createMBeanServer";
   permission javax.management.MBeanPermission "*", "*";
   permission javax.management.MBeanTrustPermission "register";
   permission java.lang.management.ManagementPermission "monitor";
   permission java.lang.management.ManagementPermission "control";

   permission java.lang.RuntimePermission "setIO";
   permission java.io.FilePermission      "jgdi.stdout", "write";
   permission java.io.FilePermission      "jgdi.stderr", "write";
   permission java.io.FilePermission      "jgdi0.log.lck", "delete";
   permission java.io.FilePermission      "${com.sun.grid.jgdi.sgeRoot}/${com.sun.grid.jgdi.sgeCell}/common/jmx/*", "read";
   permission java.io.FilePermission      "${com.sun.grid.jgdi.sgeRoot}/lib/-", "read";
   permission java.lang.RuntimePermission "accessClassInPackage.sun.management.jmxremote";
   permission java.lang.RuntimePermission "accessClassInPackage.sun.management.resources";
   permission java.lang.RuntimePermission "accessClassInPackage.sun.management";
   permission java.lang.RuntimePermission "accessClassInPackage.sun.rmi.server";
   permission java.lang.RuntimePermission "accessClassInPackage.sun.management.snmp.util";
   permission java.lang.RuntimePermission "accessClassInPackage.sun.rmi.registry";

   permission java.util.PropertyPermission "java.rmi.server.randomIDs", "write";

   permission javax.security.auth.AuthPermission "modifyPrincipals";
   permission javax.security.auth.AuthPermission "createLoginContext.*";
   permission javax.security.auth.AuthPermission "createLoginContext.JMXPluggableAuthenticator";
   permission java.security.SecurityPermission "createAccessControlContext";

   permission javax.management.remote.SubjectDelegationPermission "javax.management.remote.JMXPrincipal.controlRole";
};

grant principal javax.management.remote.JMXPrincipal "controlRole" {
   permission javax.management.MBeanPermission "com.sun.grid.jgdi.management.mbeans.JGDIJMX#*", "*";
   permission javax.management.MBeanPermission "sun.management.*#*", "*";
   permission javax.security.auth.AuthPermission "createLoginContext.jgdi";
   permission javax.security.auth.AuthPermission "doAs";
   permission javax.security.auth.AuthPermission "getSubject";
   permission java.util.PropertyPermission "*", "read";
   permission java.util.PropertyPermission "user.timezone", "read,write";
   permission java.util.logging.LoggingPermission "control";
   permission java.io.FilePermission      "${com.sun.grid.jgdi.sgeRoot}/lib/-", "read";
   permission java.lang.management.ManagementPermission "monitor";
   permission java.net.SocketPermission "*", "resolve";

   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#-[java.lang:type=OperatingSystem]", "isInstanceOf";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#-[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#ProcessCpuTime[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#Name[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#Version[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#Arch[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#AvailableProcessors[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#CommittedVirtualMemorySize[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#TotalPhysicalMemorySize[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#FreePhysicalMemorySize[java.lang:type=OperatingSystem]", "getAttribute";   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#TotalSwapSpaceSize[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#FreeSwapSpaceSize[java.lang:type=OperatingSystem]", "getAttribute";
   permission javax.management.MBeanPermission "javax.management.MBeanServerDelegate#-[JMImplementation:type=MBeanServerDelegate]", "addNotificationListener";
   permission javax.management.MBeanPermission "javax.management.MBeanServerDelegate#-[JMImplementation:type=MBeanServerDelegate]", "isInstanceOf";
   permission javax.management.MBeanPermission "javax.management.MBeanServerDelegate#-[JMImplementation:type=MBeanServerDelegate]", "getMBeanInfo";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#-[java.lang:type=OperatingSystem]", "queryNames";
   permission javax.management.MBeanPermission "java.util.logging.Logging#-[java.util.logging:type=Logging]", "queryNames";
   permission javax.management.MBeanPermission "javax.management.MBeanServerDelegate#-[JMImplementation:type=MBeanServerDelegate]", "queryNames";
   permission javax.management.MBeanPermission "java.util.logging.Logging#-[java.util.logging:type=Logging]", "isInstanceOf";
   permission javax.management.MBeanPermission "java.util.logging.Logging#-[java.util.logging:type=Logging]", "getMBeanInfo";
   permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#-[java.lang:type=OperatingSystem]", "getMBeanInfo";

};

grant {
   permission java.util.logging.LoggingPermission "control";
   permission java.util.PropertyPermission "*", "read";
   permission java.util.PropertyPermission "user.timezone", "write";
   permission java.lang.RuntimePermission "setIO";
   permission java.lang.RuntimePermission "loadLibrary.jgdi";
   permission java.io.FilePermission      "jgdi.stdout", "write";
   permission java.io.FilePermission      "jgdi.stderr", "write";
   permission java.io.FilePermission      "${com.sun.grid.jgdi.sgeRoot}/lib/-", "read";
   permission java.io.FilePermission      "${com.sun.grid.jgdi.sgeRoot}/util/arch", "execute";
   permission java.io.FilePermission      "${com.sun.grid.jgdi.sgeRoot}/utilbin/-", "execute";
   permission javax.security.auth.AuthPermission "modifyPrincipals";
   permission java.io.FilePermission "${com.sun.grid.jgdi.caTop}", "read";
   permission java.io.FilePermission "${com.sun.grid.jgdi.caTop}/cacert.pem", "read";
   permission java.io.FilePermission "${com.sun.grid.jgdi.caTop}/ca-crl.pem", "read";
   permission java.io.FilePermission "${com.sun.grid.jgdi.caTop}/usercerts/-", "read";
   permission java.io.FilePermission "${com.sun.grid.jgdi.serverKeystore}", "read";
};

/*
grant {
   permission java.security.AllPermission;
};
*/

management.properties

This file describes the general JMX server configuration and the default template looks similar to this example and is usually adapted automatically during the installation process replacing the @@SGE_*@@ variables by concrete values.
The meaning of the @@SGE_*@@ variables is:

  • @@SGE_JMX_PORT@@ is the configured JMX port
  • @@SGE_JMX_SSL@@ is true or false if SSL shall be enabled for JMX or not
  • @@SGE_JMX_SSL_CLIENT@@ is true or false if client authentication is required
  • @@SGE_JMX_SSL_KEYSTORE@@ the keystore used for enabled SSL
  • @@SGE_JMX_SSL_KEYSTORE_PW@@ the corresponding keystore password
  • @@SGE_ROOT@@ the $SGE_ROOT root directory
  • @@SGE_CELL@@ the $SGE_CELL name usually 'default'
#####################################################################
#  Default Configuration File for JGDI JMX
#####################################################################
#
# The Management Configuration file (in java.util.Properties format)
# will be read if one of the following system properties is set:
#    -Dcom.sun.grid.jgdi.management.jmxremote.port=<port-number>
# or -Dcom.sun.grid.jgdi.management.config.file=<this-file>
#
# The default Management Configuration file is:
#
#       $SGE_ROOT/{$SGE_CELL|default}/common/jmx/management.properties
#
# ################ Management Agent Port #########################
#
# For setting the JMX RMI agent port use the following line
# com.sun.grid.jgdi.management.jmxremote.port=<port-number>
com.sun.grid.jgdi.management.jmxremote.port=@@SGE_JMX_PORT@@

#####################################################################
#        RMI Management Properties
#####################################################################
#
# If system property -Dcom.sun.grid.jgdi.management.jmxremote.port=<port-number>
# is set then
#     - A MBean server is started
#     - JRE Platform MBeans are registered in the MBean server
#     - RMI connector is published  in a private readonly registry at
#       specified port using a well known name, "jmxrmi"
#     - the following properties are read for JMX remote management.
#
# The configuration can be specified only at startup time.
# Later changes to above system property (e.g. via setProperty method),
# this config file, the password file, or the access file have no effect to the
# running MBean server, the connector, or the registry.
#

#
# ###################### RMI SSL #############################
#
# com.sun.grid.jgdi.management.jmxremote.ssl=true|false
#      Default for this property is true. (Case for true/false ignored)
#      If this property is specified as false then SSL is not used.
#

#For RMI monitoring without SSL use the following line
# com.sun.grid.jgdi.management.jmxremote.ssl=false
com.sun.grid.jgdi.management.jmxremote.ssl=@@SGE_JMX_SSL@@

# com.sun.grid.jgdi.management.jmxremote.ssl.enabled.cipher.suites=<cipher-suites>
#      The value of this property is a string that is a comma-separated list
#      of SSL/TLS cipher suites to enable. This property can be specified in
#      conjunction with the previous property "com.sun.management.jmxremote.ssl"
#      in order to control which particular SSL/TLS cipher suites are enabled
#      for use by accepted connections. If this property is not specified then
#      the SSL RMI Server Socket Factory uses the SSL/TLS cipher suites that
#      are enabled by default.
#

# com.sun.grid.jgdi.management.jmxremote.ssl.enabled.protocols=<protocol-versions>
#      The value of this property is a string that is a comma-separated list
#      of SSL/TLS protocol versions to enable. This property can be specified in
#      conjunction with the previous property "com.sun.management.jmxremote.ssl"
#      in order to control which particular SSL/TLS protocol versions are
#      enabled for use by accepted connections. If this property is not
#      specified then the SSL RMI Server Socket Factory uses the SSL/TLS
#      protocol versions that are enabled by default.
#

# com.sun.grid.jgdi.management.jmxremote.ssl.need.client.auth=true|false
#      Default for this property is false. (Case for true/false ignored)
#      If this property is specified as true in conjunction with the previous
#      property "com.sun.management.jmxremote.ssl" then the SSL RMI Server
#      Socket Factory will require client authentication.
#

#For RMI monitoring with SSL client authentication use the following line
#com.sun.grid.jgdi.management.jmxremote.ssl.need.client.auth=true
com.sun.grid.jgdi.management.jmxremote.ssl.need.client.auth=@@SGE_JMX_SSL_CLIENT@@


#
# ################ RMI User authentication ################
#
# com.sun.grid.jgdi.management.jmxremote.authenticate=true|false
#      Default for this property is true. (Case for true/false ignored)
#      If this property is specified as false then no authentication is
#      performed and all users are allowed all access.
#

# For RMI monitoring without any checking use the following line
# com.sun.grid.jgdi.management.jmxremote.authenticate=false
com.sun.grid.jgdi.management.jmxremote.authenticate=true

#
# ################ RMI Login configuration ###################
#
# com.sun.grid.jgdi.management.jmxremote.login.config=<config-name>
#      Specifies the name of a JAAS login configuration entry to use when
#      authenticating users of RMI monitoring.
#
#      Setting this property is optional - the default login configuration
#      specifies a file-based authentication that uses the password file.
#
#      When using this property to override the default login configuration
#      then the named configuration entry must be in a file that gets loaded
#      by JAAS. In addition, the login module(s) specified in the configuration
#      should use the name and/or password callbacks to acquire the user's
#      credentials. See the NameCallback and PasswordCallback classes in the
#      javax.security.auth.callback package for more details.
#
#      If the property "com.sun.management.jmxremote.authenticate" is set to
#      false, then this property and the password & access files are ignored.
#

# For a non-default login configuration use the following line
# com.sun.grid.jgdi.management.jmxremote.login.config=<config-name>
com.sun.grid.jgdi.management.jmxremote.login.config=GridwareConfig

#
# ################ RMI Password file location ##################
#
# com.sun.grid.jgdi.management.jmxremote.password.file=filepath
#      Specifies location for password file
#      This is optional - default location is
#      $JRE/lib/management/jmxremote.password
#
#      If the property "com.sun.grid.jgdi.management.jmxremote.authenticate" is set to
#      false, then this property and the password & access files are ignored.

# For a non-default password file location use the following line
# com.sun.grid.jgdi.management.jmxremote.password.file=filepath
com.sun.grid.jgdi.management.jmxremote.password.file=@@SGE_ROOT@@/@@SGE_CELL@@/common/jmx/jmxremote.password

#
# ################ RMI Access file location #####################
#
# com.sun.grid.jgdi.management.jmxremote.access.file=filepath
#      Specifies location for access  file
#      This is optional - default location is
#      $JRE/lib/management/jmxremote.access
#
#      If the property "com.sun.management.jmxremote.authenticate" is set to
#      false, then this property and the password & access files are ignored.
#      Otherwise, the access file must exist and be in the valid format.
#      If the access file is empty or non-existent then no access is allowed.
#

# For a non-default access file location use the following line
# com.sun.grid.jgdi.management.jmxremote.access.file=filepath
com.sun.grid.jgdi.management.jmxremote.access.file=@@SGE_ROOT@@/@@SGE_CELL@@/common/jmx/jmxremote.access


# For the JGDI keystore module use this settings for the server keystore and keystore password
com.sun.grid.jgdi.management.jmxremote.ssl.serverKeystore=@@SGE_JMX_SSL_KEYSTORE@@
com.sun.grid.jgdi.management.jmxremote.ssl.serverKeystorePassword=@@SGE_JMX_SSL_KEYSTORE_PW@@

jmx.access

The jmx access file defines which principals are mapped to a special role.

######################################################################
#     Default Access Control File for Remote JMX(TM) Monitoring
######################################################################
#
# Access control file for Remote JMX API access to monitoring.
# This file defines the allowed access for different roles.  The
# password file (jmxremote.password by default) defines the roles and their
# passwords.  To be functional, a role must have an entry in
# both the password and the access files.
#
# Default location of this file is $JRE/lib/management/jmxremote.access
# You can specify an alternate location by specifying a property in
# the management config file $JRE/lib/management/management.properties
# (See that file for details)
#
# The file format for password and access files is syntactically the same
# as the Properties file format.  The syntax is described in the Javadoc
# for java.util.Properties.load.
# Typical access file has multiple  lines, where each line is blank,
# a comment (like this one), or an access control entry.
#
# An access control entry consists of a role name, and an
# associated access level.  The role name is any string that does not
# itself contain spaces or tabs.  It corresponds to an entry in the
# password file (jmxremote.password).  The access level is one of the
# following:
#       "readonly" grants access to read attributes of MBeans.
#                   For monitoring, this means that a remote client in this
#                   role can read measurements but cannot perform any action
#                   that changes the environment of the running program.
#       "readwrite" grants access to read and write attributes of MBeans,
#                   to invoke operations on them, and to create or remove them.
#         This access should be granted to only trusted clients,
#                   since they can potentially interfere with the smooth
#         operation of a running program
#
# A given role should have at most one entry in this file.  If a role
# has no entry, it has no access.
# If multiple entries are found for the same role name, then the last
# access entry is used.
#
#
# Default access control entries:
# o The "monitorRole" role has readonly access.
# o The "controlRole" role has readwrite access.

monitorRole   readonly
controlRole   readwrite

jmx.password

This is also a possible simple authentication mechanism though not recommended. Usually the jaas login module is preferred since it is much more flexible. You can specify a password for the different roles there. If a simple login mechanism is required it is recommended to change management.properties to use TestConfig instead of GridwareConfig, which allows any valid Unix user to connect to JGDI JMX server without a password.

logging.properties

To enable JGDI and JMX logging the delivered logging file has to be adjusted and sge_qmaster or at least the JMX server has to be restarted. The generated logging files default to jgdi0.log, jgdi.stderr and jgdi.stdout in the master spooling directory. The logging can also be influenced by changing the additional_jvm_args configuration to enable additional debugging messages for example.

#
#  Java Logging Configuration for JMX MBean server
#

# Specify the handlers to create in the root logger
# (all loggers are children of the root logger)
# The following creates two handlers

# Per default we log to the console
#handlers = java.util.logging.ConsoleHandler

# Use FileHandler
handlers = java.util.logging.FileHandler

# ------------------------------------------------------------------------------
#  Definition of log levels
# ------------------------------------------------------------------------------
# Set the default logging level for the root logger
.level = INFO
#com.sun.grid.jgdi.JGDI.level = FINE
#com.sun.grid.jgdi.rmi.level = FINE
#com.sun.grid.jgdi.configuration.xml.XMLUtil.level = FINE
#com.sun.grid.jgdi.configuration.ClusterQueueTestCase.level = FINE
#com.sun.grid.jgdi.management.level = FINER
#com.sun.grid.jgdi.event.level = FINER
# For authuser login module debugging
#com.sun.grid.security.login.level = FINER
#com.sun.grid.util.expect.level = FINER

# ------------------------------------------------------------------------------
#  Settings for ConsoleHandler
# ------------------------------------------------------------------------------
# Set the default logging level for new ConsoleHandler instances
java.util.logging.ConsoleHandler.level = INFO

# Set the default formatter for new ConsoleHandler instances
java.util.logging.ConsoleHandler.formatter = com.sun.grid.jgdi.util.SGEFormatter

# ------------------------------------------------------------------------------
#  Settings for FileHandler
# ------------------------------------------------------------------------------
# Set the default logging level for new FileHandler instances
java.util.logging.FileHandler.level = ALL
# qmaster runs in qmaster spool dir, so the file is created there
java.util.logging.FileHandler.pattern=jgdi%u.log
java.util.logging.FileHandler.formatter=com.sun.grid.jgdi.util.SGEFormatter

#
# Possible columns:
#
#   time    timestamp of the log message
#   host    hostname of the log message
#   name    name of the logger
#   thread  id of the thread
#   level   log level (short form)
#   source  class and method name
#   level_long log_level long form
#
com.sun.grid.jgdi.util.SGEFormatter.columns = time thread source level message

#
#  Print the stacktrace of the log record
#
com.sun.grid.jgdi.util.SGEFormatter.withStacktrace=true

#
#  Delimiter between columns
#
com.sun.grid.jgdi.util.SGEFormatter.delimiter = |

Testing and Troubleshooting

To connect to the JMX server jconsole can be used for testing. It is the responsibility of the administrator to allow/disallow access to the system via JMX. To force also client authentication of jconsole the management.properties file must be configured with:

  • com.sun.grid.jgdi.management.jmxremote.ssl=true
  • com.sun.grid.jgdi.management.jmxremote.ssl.need.client.auth=true
% jconsole -J-Djava.security.manager=java.rmi.RMISecurityManager \
 -J-Djava.security.policy=$SGE_ROOT/util/rmiconsole.policy \
 -J-Djavax.net.ssl.trustStore=<server truststore> \
 [-J-Djavax.net.ssl.keyStore=/<safe>/mykeystore \
  -J-Djavax.net.ssl.keyStorePassword=<mykeystore_pw> \
  -J-Djavax.net.ssl.keyPassword=<mykeystore_pw> ] \
 [-J-Djavax.net.debug=ssl]

where <server truststore> usually is either:
/var/sgeCA/port5322/$SGE_CELL/private/keystore
(only the server certificate is accessible without password)
or a special truststore is made available by the administrator:

keytool -export -alias "root" \ 
        -keystore /var/sgeCA/port$SGE_QMASTER_PORT/$SGE_CELL/private/keystore -rfc -file /tmp/jmxserver.cer

keytool -import -file /tmp/jmxserver.cer -keystore /tmp/truststore
Enter keystore password:  <pwd>
...
Trust this certificate? [no]:  yes
Certificate was added to keystore

The optional arguments are required if client authentication is set to true or for debugging.

The following simple example can be used to connect via JMX and monitor events

% java [-Dcom.sun.grid.jgdi.keyStore=\          /var/sgeCA/port$SGE_QMASTER_PORT/$SGE_CELL/private/keystore \
-Dcom.sun.grid.jgdi.caTop="$SGE_ROOT/$SGE_CELL/common/sgeCA" \
-Djava.util.logging.config.file=util/shell_logging.properties ] \
-cp $SGE_ROOT/lib/juti.jar:$SGE_ROOT/lib/jgdi.jar \
com.sun.grid.jgdi.examples.jmxeventmonitor.Main

The optional arguments can be skipped and serve only to preset the login dialog with useful values. If a connection has been established once a preferences file is written, that is reused afterwards.
To have the correct environment variables set the SGE settings.(c)sh file has to be sourced. To get access to the keystore the command must be run by the admin user in the example above.

For troubleshooting the following settings and files might give some additional insights:

  • Messages file in the master spool directory if the JMX server can't be started
  • $SGE_ROOT/$SGE_CELL/common/bootstrap to check if jvm_threads is enabled at all
  • jgdi* log files in the master spool directory are the main source for finding out the reason for failure analysis
  • $SGE_ROOT/$SGE_CELL/common/jmx/logging.properties to enable more detailed logging
  • qconf -mconf with an additional_jvm_args parameter
    For example, add these two arguments -Djava.security.debug=all -Djavax.net.debug=ssl to trace any permission and authentication problems.
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Sun Microsystems community and they might not be employed or in any way formally affiliated with Sun Microsystems. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Sun nor any other party necessarily agrees with them.

Copyright 1994-2009 Sun Microsystems, Inc.
Powered by Atlassian Confluence Sun Guidelines on Public Discourse Privacy Policy Terms of Use Trademarks Site Map Employment Investor Relations Contact