Installing ARCo

Searching Sun Grid Engine 6.2

Sun Grid Engine Information Center
Accounting and Reporting Console (ARCo)
Index

Installing ARCo

Installing the Accounting and Reporting console software (ARCo) consists of the following topics:

Planning the ARCo Installation

Before you install the ARCo software, you must plan how to achieve the results that fit your environment. This section helps you make the decisions that affect the rest of the procedure. Write down your installation plan in a table similar to the following example.

Parameter      Value     
sge-root directory  
Database software vendor  
Database user (read access)  
Database user (write access)  
Multi-cluster support?  

Supported Operating Platforms

  • Solaris 10, 9, and 8 Operating Systems (SPARC Platform Edition)
  • Solaris 10 and 9 Operating Systems (x86 Platform Edition)
  • Solaris 10 Operating System (x64 Platform Edition)
  • Linux x64, kernel 2.4, 2.6, glibc >= 2.3.2
  • Linux x86, kernel 2.4, 2.6, glibc >= 2.3.2

Required Software

For ARCo software to function correctly, you must already have installed the following on your ARCo system:

  • Sun Grid Engine 6.2 software
  • Java Runtime Environment (JRE) version 1.5
  • One of the following database software versions
    • PostgresSQL 8.0 through 8.3
    • MySQL 5.0.36 and higher
    • Oracle 9i or 10g
  • Sun Java Web Console version 3.0 and one of the following web browsers:
    • Netscape 6.2 and above
    • Mozilla 1.4 and above
    • Internet Explorer 5.5 and above
    • Firefox 1.0 and above
Note
Sun Java Web Console 3.0 is installed automatically with Solaris 10 Update 3 or later. If you need to install Sun Java Web Console, see Installing Sun Java Web Console.

Disk Space Recommendations

Component Space Needed
ARCo software 100 MB
Sun Java Web Console ??
Database server memory 250 to 750 MB *
Database server disk space 10 GB *

* Your specific database server configuration settings depend on the following:

  • Cluster size and number of jobs running on cluster
  • Setting of joblog in reporting_params (qconf -mconf)
  • Configured report_variables (qconf -me global)
  • Configuration of dbwriter deletion rules (<sge_root>/dbwriter/database/<database_type>/dbwriter.xml)

For guidelines about determining specific database needs, see Space Requirements for the ARCo Database on the Open Grid Engine site.

Multi-Cluster Support Overview

If you have multiple Sun Grid Engine clusters, you can log in to one instance of ARCo from which you can run reports on all ARCo instances that use the same database vendor and structure. With the ARCo multi-cluster support, one dbwriter instance per qmaster is still required, but a single reporting installation is sufficient for all qmasters. During the reporting installation, you can supply separate database parameters, such as database name, database user, database host, database password for each cluster, the only condition being that that databases are of the same vendor. Database connections are configured from these parameters, which enables you to run the same queries on separate clusters, while logged in to the single instance of the ARCo reporting.

For the multi-cluster database configuration, you can use any of these database setups:

  • A single database with multiple schemas (one per each cluster) on a single DBMS
  • Separate databases (one per each cluster) on a single DBMS
  • Separate databases on separate DBMS (one per each cluster)

If you are not interested in cross-cluster queries, you can choose any of these setups. However, to run cross-cluster queries, you must configure a single database with multiple schemas (one per each cluster) on a single DBMS.

Database Configuration Illustrations

The following diagrams illustrate the supported database configurations. Additional steps, described in How to Configure the ARCo Database with Multiple Schemas on PostgreSQL, are necessary to configure a PostgresSQL database with separate schemas.

If you want to run cross-cluster queries, use the configuration depicted in One Database With Multiple Schemas on a Single DBMS. Otherwise, you can choose either of the other configurations, although Separate Databases on a Single DBMS is slightly preferred.

Figure - Separate Databases on Separate DBMS

Each database is created on a separate Database Management Server (DBMS).

"Diagram shows two databases

Figure - Separate Databases on a Single DBMS

Databases of different names are created on the same DBMS. Only two users are required to access all ARCo databases on the same server, arco_read and arco_write.

Diagram shows two databases on one DBMS."

Figure - One Database With Multiple Schemas on a Single DBMS

Only one database is created with multiple schemas (one per each cluster). There are two users for each schema, arco_write_cluster and arco_read_cluster. The name of the schema should be the same as the name of the owner (arco_write_cluster).

Diagram shows one database on one DBMS with multiple schemas.

Schema Overview

A database cluster contains one or more named databases. Any given client connection to the server can access only the data in a single database, the one specified in the connection request. A database can contain one or more named schemas, which in turn contain tables. Schemas also contain other objects, such as views, aliases, indexes and functions. The same object name can be used in different schemas without conflict; for example, both schemas arco_write_denver and arco_write_london may contain the sge_job table.

Unlike databases, schemas are not rigidly separated: a user may access objects in any of the schemas in the database to which the user is connected, if the user has privileges to do so. For user to access objects from a different schema, he needs to be granted SELECT privilege on the objects and access them using the fully-qualified name, for example schema_name.table_name. A user does not need to use the fully-qualified names if accessing objects in its own schema.

Each database handles the schema notion differently.

  • Oracle - In Oracle, one schema is created automatically for each user. Because there is a 1-to-1 relationship between a user and a schema, these two terms are often used interchangeably. To perform cross-cluster queries, one designated database user (e.g multi_read) needs to be granted SELECT privileges on all the objects (tables, views) from all the other schemas. See Using Oracle Database
  • PostgreSQL - In PostgreSQL when a table is created without explicitly specifying schema name, it is automatically put into the default schema called 'public'. Each PostgreSQL database contains such a schema and all users have ALL privilege on that schema. The dbdefinition.xml for Postgres allows for explicit specification of schema for table definition. Detailed instructions, described in How to Configure the ARCo Database with Multiple Schemas on PostgreSQL. For more information on schemas, see http://www.postgresql.org/docs/8.2/static/ddl-schemas.html.
  • MySQL – MySQL does not support schemas; the term schema is analogous to database. Command CREATE SCHEMA is the same as CREATE DATABASE. The implication is that a user can access any object from any database on the same DBMS using a client connection to any single database. If you configure MySQL multiple databases using just one pair of users (arco_write, arco_read) and grant the privileges as described in step 7 of How to Configure the ARCo Database on MySQL, you can perform cross_cluster queries. You must use the fully-qualified names when accessing objects, for example database_name.table_name.

Configuring the Database Server

You must properly install and configure the database server before you can install and use ARCo. Specific database installation instructions and configuration settings differ by database vendor. For more information, see the following:

How to Configure the PostgresSQL Server

Before you configure the database server, you must download, compile and install the PostgreSQL database software and create a user account to own the database processes. Usually, this user is postgres. Add the PostgreSQL bin directory and necessary LD_LIBRARY_PATH settings to your environment. You can find detailed information on the PostgreSQL database in the Postgres documentation.

  1. Create a home directory for the postgres user.
    In this example, the home directory is /space/postgres/data. 
    % mkdir -p /space/postgres/data
% useradd -d /space/postgres  postgres 
% chown postgres /space/postgres/data
% su - postgres
    • Continue as described in the PostgreSQL documentation to set up a database.* 
      > initdb -D /space/postgres/data

creating directory /space/postgres/data... ok
creating directory /space/postgres/data/base... ok
creating directory /space/postgres/data/global... ok
creating directory /space/postgres/data/pg_xlog... ok
creating directory /space/postgres/data/pg_clog... ok
creating template1 database in /space/postgres/data/base/1... ok
creating configuration files... ok
initializing pg_shadow... ok
enabling unlimited row size for system tables... ok
initializing pg_depend... ok
creating system views... ok
loading pg_description... ok
creating conversions... ok
setting privileges on built-in objects... ok
vacuuming database template1... ok
copying template1 to template0... ok

Success. You can now start the database server using:
   postmaster -D /space/postgres/data
   or
   pg_ctl -D /space/postgres/data -l logfile start


  2. Make the following changes to the pg_hba.conf file.
    This change permits unrestricted and password free access to the database superuser postgres but requires md5 encrypted passwords for all other database users. Replace nnn.nnn.nnn with your subnet address without the trailing 0. You also can add access rules on a per-host basis by adding similar lines with host IP addresses. 
    # TYPE  DATABASE  USER  IP-ADDRESS     IP-MASK         METHOD
local   all       postgres                             trust 
local   all       all                                  md5 
# IPv4-style local connections: 
#host    all       all   nnn.nnn.nnn.0  255.255.255.0  md5


  3. Make the following changes to the postgresql.conf file, to enable TCP/IP access from other hosts. 
    tcpip_socket = true
max_connections = 40   (increase if necessary)
    Note
    Ensure that the value of shared_buffers is at least twice the value of max_connections. On PostgreSQL > 8.0 also modify the value of listen_addresses


  4. Start the database.
    In this example, -i enables TCP/IP communication, while -S is for silent mode. 
    > postmaster -S -i


  5. Verify the installation.
    As the postgres user, try the following commands: 
    % su - postgres
> createuser -P test_user
Enter password for new user: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new user be allowed to create databases? (y/n) y
Shall the new user be allowed to create more new users? (y/n) n
CREATE USER
> createdb -O test_user -E UNICODE test
CREATE DATABASE


  6. Execute commands as the database super user. 
     > psql test
Welcome to psql 8.3, the PostgreSQL interactive terminal.
Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help on internal slash commands
       \g or terminate with semicolon to execute query
       \q to quit
test=# create table test (x int, y text);
CREATE TABLE
test=# insert into test values (1, 'one');
INSERT 16982 1
test=# insert into test values (2, 'two');
INSERT 16983 1
test=# select * from test;
x |  y
---+------
1 | one
2 | two
(2 rows)
test=# \q
 > psql -U test_user test
Password: 
Welcome to psql 8.3, the PostgreSQL interactive terminal.
Type:  \copyright for distribution terms
       \h for help with SQL commands
       \? for help on internal slash commands
       \g or terminate with semicolon to execute query
       \q to quit
test=>


  7. After you have successfully tested your database software, set up the PostgresSQL database.
  8. After you have set up the database, install the accounting and reporting software.
    See Installing dbwriter and Reporting Software.

How to Configure the ARCo Database on PostgresSQL

  1. Configure the PostgresSQL database server, as described in How to Configure the PostgresSQL Server.

  2. Log in as the database superuser, e.g. postgres. 
    # su - postgres


  3. Create the database owner name and password.
    You will need this information when you install the dbwriter and ARCo console as described in Installing dbwriter and Reporting Software. 
    > createuser -P arco_write
Enter password for new user: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new user be allowed to create databases? (y/n) y
Shall the new user be allowed to create more new users? (y/n) n
CREATE USER


  4. Create the accounting and reporting database. 
    > createdb -O arco_write arco
CREATE DATABASE


  5. Create a database user for reading the database. 
    > createuser -P arco_read
Enter password for new user: 
Enter it again: 
Shall the new role be a superuser? (y/n) n
Shall the new user be allowed to create databases? (y/n) n
Shall the new user be allowed to create more new users? (y/n) n
CREATE USER


  6. Grant arco_write permissions on default tablespace.
    The dbdefinition.xml explicitly specifies tablespace name in table definition.
    The arco_write must have permissions to create objects in the specified tablespace. 
    > psql

postgres=# GRANT CREATE ON TABLESPACE pg_default TO arco_write;
    Note
    By using tablespaces, an administrator can control the disk layout of a database installation and optimize performance.
    You can find detailed information on the PostgreSQL tablespaces in the Postgres documentation.


  7. After you have set up the database, install the accounting and reporting software.
    See Installing dbwriter and Reporting Software.

How to Configure the ARCo Database with Multiple Schemas on PostgreSQL

  1. Configure the PostgresSQL database server, as described in How to Configure the PostgresSQL Server.

  2. Log in as the database superuser, e.g. postgres. 
    # su - postgres


  3. Create database user for writing. 
    > createuser -P arco_write_london

 Enter password for new user:  
 Enter it again:  
 Shall the new role be a superuser? (y/n) n
 Shall the new user be allowed to create databases? (y/n) y 
 Shall the new user be allowed to create more new users? (y/n) n
 CREATE USER


  4. Repeat step 2 for each cluster, changing the user name.
    For example, if you have a second cluster called denver, you might use arco_write_denver.
    Note
    The user, database, schema names are arbitrary. You are free to use your own, these were chosen for the demonstrative purposes.


  5. Create a database user for reading. 
    > createuser -P arco_read_london
 Enter password for new user:  
 Enter it again:  
 Shall the new role be a superuser? (y/n) n
 Shall the new user be allowed to create databases? (y/n) n 
 Shall the new user be allowed to create more new users? (y/n) n
 CREATE USER


  6. Repeat step 4 for each cluster, changing the user name.
    For example, if you have a second cluster called denver, you might use arco_write_denver.

  7. Create the accounting and reporting database. 
    > createdb arco

CREATE DATABASE


  8. Log in to the accounting and reporting database console. 
    > psql arco

 arco=#


  9. Grant arco_write_london permissions on default tablespace.
    The dbdefinition.xml explicitly specifies tablespace name in table definition.
    The arco_write must have permissions to create objects in the specified tablespace. 
    arco=# GRANT CREATE ON TABLESPACE pg_default TO arco_write_london;
    Note
    By using tablespaces, an administrator can control the disk layout of a database installation and optimize performance.
    You can find detailed information on the PostgreSQL tablespaces in the Postgres documentation.


  10. Repeat step 8 for each cluster, changing the user name.

  11. Create schemas.
    The schema name should equal the owner name of the schema. The owner of the schema is the arco_write_cluster user. 
    arco=# CREATE SCHEMA arco_write_london AUTHORIZATION arco_write_london; 
CREATE SCHEMA

    Schema arco_write_london owned by user arco_write_london was created.

  12. Repeat step 10 for each cluster, changing the schema name and owner name.

  13. Grant appropriate privileges for users to schemas that they do not own.
    By default, users cannot access any objects in schemas they do not own. To allow other user access to the schema, the user needs to be granted USAGE privilege on that schema. Grant arco_read_cluster the USAGE privilege on the arco_write_cluster schema. 
    arco=# GRANT USAGE ON SCHEMA arco_write_london TO arco_read_london;
GRANT


  14. Repeat step 12 for each cluster, changing the schema name and arco_read_cluster name.
    For example, for a Denver cluster the schema name should be arco_write_denver and the user should be arco_read_denver.

  15. Set search path for ARCo users.
    In the reporting queries, tables are referred to by unqualified names, which consist of just the table name. The system determines which table is meant by following a search path, which is a list of schemas to look in. The first matching table in the search path is taken to be the one wanted. If there is no match in the search path, an error is reported, even if matching names exist in other schemas in the database.

    In a default setup the search path is: $user, public
    Command SHOW search_path; can be run to show search path for the currently connected user.

    If the schemas in step 10. were created using the pattern schema_name = user_name, then no additional steps are required for arco_write_cluster users. The arco_read_cluster needs to be altered. 
    arco=# ALTER USER arco_read_london SET search_path=arco_write_london;
ALTER ROLE


  16. Repeat step 14 for each cluster, changing the arco_read_cluster and the search_path.

  17. Verify that search_paths are set correctly. 
    arco=# SELECT * FROM pg_user;

    Each arco_read_cluster user should have search_path in useconfig column set to the appropriate arco_write_cluster. Each arco_write_cluster user should have the useconfig field empty, signifying the default search_path.

  18. Create the cross-cluster user
    Note
    In order to perform cross-cluster queries, one user has to be granted SELECT privileges an all the objects in all of the schemas and access these objects using the fully-qualified name, for example <schema_name>.<table_name>. For clarity, we will create a new user. However, you can choose any of your existing users. You will need to supply information for this user during the installation of the reporting module.

    Perform steps 13 - 18 of Migrating PostgreSQL Database to a Different Schema.
    See the example of a cross-cluster query.

  19. After you have set up the database, install the dbwriter and reporting software.
    See Installing dbwriter and Reporting Software.

How to Configure the MySQL Database Server

The Accounting and Reporting Console uses views. As a result, the console supports MySQL database version 5.0.36 and higher. For more information on the MySQL database software, see the MySQL documentation.

Case Sensitivity in MySQL Database

In MySQL, databases correspond to directories within the data directory. Each table within a database corresponds to at least one file within the database directory. Because of this the case sensitivity of the underlying operating system determines the case sensitivity of database and table names. Therefore, database and table names are case sensitive in most varieties of UNIX, and not case sensitive in Windows.

  1. Download the appropriate MySQL software for your system from http://www.mysql.com.
    The standard installation directory for UNIX systems is /usr/local/mysql. If you install the software into a different directory, you have to change the settings for the scripts provided in the package.
    Note
    ARCo is a Java web-based application and needs the Java DataBase Connectivity (JDBC) driver for converting JDBC calls into the network protocol used by the MySQL database. You can download the JDBC driver from http://www.mysql.com/products/connector.


  2. Create a symbolic link from the installation directory to MySQL. 
    # ln -s $installation_directory/mysql-standard-5.0.26-solaris10-i386 mysql

    The mysql directory contains several files and subdirectories.

  3. Add a login user and group for mysqld. 
    # groupadd mysql
# useradd -g mysql mysql


  4. Create the MySQL grant tables. 
    # scripts/mysql_install_db --user=mysql


  5. Change the ownership of program binaries to root and ownership of the data directory to the user that you use to run mysqld. 
    # chown -R root .
# chown -R mysql data
# chgrp -R mysql .


  6. Configure MySQL server to use InnoDB as the default storage engine.
    MySQL supports several storage engines that act as handlers for different table types. MySQL storage engines include both, those that handle transaction-safe tables, and those that handle non-transaction-safe tables. ARCo installation requires the use of transaction-safe tables.

    Edit the my.cnf file and set the option 
    default_storage_engine=innodb

    Configure other innodb properties such as innodb_data_home_dir, innodb_data_file_path.

    For details on InnoDB storage configuration see: http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html

  7. Start the MySQL server. 
    # bin/mysqld_safe --user=mysql &amp;


  8. Assign the root password. 
    # ./bin/mysqladmin -u root password 'new-password'
# ./bin/mysqladmin -u root -h ${hostname} password 'new-password'


  9. Verify installation.
    Log in to the MySQL console as a superuser. 
    # mysql -u root -p<password>

    As a superuser perform these commands:

    mysql> GRANT ALL on *.* to 'test'@'<database_host>' identified by '<password>' with GRANT OPTION;
mysql> GRANT ALL on *.* to 'test'@'%' identified by '<password>' with GRANT OPTION;

    Log out and log in as the user test.

    mysql> \q
# mysql -u test -p<password>

    As the user test perform these commands:

    mysql> CREATE DATABASE test;
mysql> USE test;
mysql> CREATE TABLE sge_test (x integer, y varchar(50));
mysql> SHOW TABLE STATUS FROM test LIKE 'sge_test';


    Important: The field 'Engine' should have a value InnoDB.

MySQL Installation Tips
  • To start the MySQL server at boot time, copy support-files/mysql.server to /etc/init.d and link it to both /etc/rc3.d/S99mysql and /etc/rc0.d/K01mysql.
    If MySQL is not installed in /usr/local/mysql, edit the file to change the basedir and datadir variables.
  • Add the full pathname of this directory to your PATH environment variable so that your shell finds the MySQL programs properly.

How to Configure the ARCo Database on MySQL

  1. Log in to the MySQL console as a superuser. 
    # mysql -u root -p<password>


  2. Create user arco_write and grant him privileges. 
    mysql> GRANT ALL on *.* to 'arco_write'@'<database_host>' identified by '<password>' with GRANT OPTION;
mysql> GRANT ALL on *.* to 'arco_write'@'%' identified by '<password>' with GRANT OPTION;


  3. Exit the MySQL console. 
    mysql> \q


  4. Log in to the MySQL console as arco_write user. 
    # mysql -u arco_write -p<password>


  5. Create the accounting and reporting database. 
    mysql> CREATE DATABASE <db_name>;


  6. Create user arco_read and grant him privileges. 
    mysql> GRANT SELECT,SHOW VIEW on <db_name>.* to 'arco_read'@'<database_host>' identified by '<password>';
mysql> GRANT SELECT,SHOW VIEW on <db_name>.* to 'arco_read'@'%' identified by '<password>';
    Note
    The user 'arco_read'@'%' must be created. If the MySQL database host is the same as the host Sun Java Web Console host where ARCo is running, you also need to create the user 'arco_read'@'database_host'.


  7. Multi-cluster configuration.
    If you are configuring databases for multiple clusters, repeat step 5. through 6., changing the db_name, or grant the user privileges on all databases. 
    mysql> GRANT SELECT,SHOW VIEW on *.* to 'arco_read'@'<database_host>' identified by '<password>';
mysql> GRANT SELECT,SHOW VIEW on *.* to 'arco_read'@'%' identified by '<password>';



    Since the same set of read and write users is used for all databases, no additional steps are required to perform cross-cluster queries.
    See the example of a cross-cluster query.

  8. Install the dbwriter and reporting software.
    See Installing dbwriter and Reporting Software.

Using Oracle Database

  1. Ask your database administrator for an instance of an Oracle database.
    You need two database users for this instance, arco_write and arco_read. The arco_write user must be able to create or alter tables, views, and indexes. During the installation of dbwriter, the arco_read user is granted SELECT privileges on the objects owned by the arco_write user and SYNONYMS for these objects are created in the schema of arco_read user. The SYNONYMS are created by arco_read user, so this user needs to have privilege to create synonyms.

    Here is an example how these users should be created on Oracle:
    Note

    The actual TABLESPACE and QUOTA values might differ.

    CREATE USER "ARCO_WRITE" PROFILE "DEFAULT" IDENTIFIED BY "<password>" 
       DEFAULT TABLESPACE "USERS" TEMPORARY TABLESPACE "TEMP" QUOTA 100 M ON "USERS" ACCOUNT UNLOCK;

CREATE USER "ARCO_READ" PROFILE "DEFAULT" IDENTIFIED BY "<password>" 
       DEFAULT TABLESPACE "USERS" TEMPORARY TABLESPACE "TEMP" QUOTA 100 M ON "USERS" ACCOUNT UNLOCK;

GRANT CREATE TABLE, CREATE VIEW, CREATE SESSION TO "ARCO_WRITE"
GRANT CREATE SYNONYM, CREATE SESSION TO "ARCO_READ"


  2. Multi-cluster configuration.
    If you have multiple Sun Grid Engine clusters, you will need one pair of users (arco_write_cluster, arco_read_cluster) for each cluster. You will need to install one dbwriter module per cluster, providing one pair of users each time, but only one reporting installation for all the clusters is necessary. During the installation of reporting module, you will provide information for all your clusters (database schemas).

  3. Cross-cluster queries.
    If you intend to perform cross-cluster queries, ask your database administrator to create another user (multi_read) and grant him SELECT privileges on all the objects from the other database schemas. You will provide the information for this user during the installation of the reporting module, and use it to connect to database when performing cross-cluster queries.
    See the example of a cross-cluster query.

  4. Ask your database administrator for the connection parameters to the database.

  5. Install the dbwriter and reporting software.
    See Installing dbwriter and Reporting Software.
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Sign up or Log in to add a comment or watch this page.


The individuals who post here are part of the extended Sun Microsystems community and they might not be employed or in any way formally affiliated with Sun Microsystems. The opinions expressed here are their own, are not necessarily reviewed in advance by anyone but the individual authors, and neither Sun nor any other party necessarily agrees with them.

Copyright 1994-2009 Sun Microsystems, Inc.
Powered by Atlassian Confluence Sun Guidelines on Public Discourse Privacy Policy Terms of Use Trademarks Site Map Employment Investor Relations Contact