Sun Cluster 3.1 9/04 Release Notes Supplement

This page supplements the standard user documentation, including the Sun Cluster 3.1 9/04 Release Notes for Solaris OS that shipped with the Sun™ Cluster 3.1 product. These “online release notes” provide the most current information on the Sun Cluster 3.1 product. This page includes the following information.

• Revision Record
• New Features
• Restrictions and Requirements
• Known Problems
• Known Documentation Problems

Top

Revision Record

The following tables list the information contained in this page and provides the revision date for this information.

Sun Cluster 3.1 9/04 Release Notes Supplement Revision Record 2006

Sun Cluster 3.1 9/04 Release Notes Supplement Revision Record 2005

Top

New Features

In addition to features documented in the Sun Cluster 3.1 9/04 Release Notes for Solaris OS, this release now includes support for the following features.

Top

SPARC: Support for VxVM 4.1 and VxFS 4.1

A patch to Sun Cluster 3.1 software adds support on Sun Clusterr 3.1 9/04 configurations and earlier for Veritas Volume Manager 4.1 and Veritas File System 4.1 software. Download and install the latest Sun Cluster 3.1 Core Patch from http://www.sunsolve.com. This support addition is associated with the bug fix for 6230506.

Top

Mirroring Internal Disks on Servers that Use Internal Hardware Disk Mirroring or Integrated Mirroring

Some servers support the mirroring of internal hard drives (internal hardware disk mirroring or integrated mirroring) to provide redundancy for node data. To use this feature in a cluster environment, follow the steps in this section.

Depending on the version of the Solaris operating system you use, you might need to install a patch to correct change request 5023670 and ensure the proper operation of internal mirroring. Check the PatchPro site to find the patch for your server.

The best way to set up hardware disk mirroring is to perform RAID configuration during cluster installation, before you configure multipathing. If you need to change your mirroring configuration after you have established the cluster, you must perform some cluster-specific steps to clean up the device IDs.

For specifics about how to configure your server's internal disk mirroring, refer to the documents that shipped with your server and the raidctl(1M) man page.

Configuring Internal Disk Mirroring During Installation

This procedure contains instructions for configuring internal disk mirroring after you have installed the Solaris OS and using the raidctl utility. Some servers might have restrictions on when you can configure internal mirroring. For the specific requirements of your server, see the documentation that shipped with your server hardware.

Before You Begin

Install your cluster hardware as instructed in your server and storage array documentation.

1. Install the Solaris operating system, as instructed in the Sun Cluster installation guide.
   
   As a part of this procedure, you will check the PatchPro web site and install any necessary patches.
   
2. Configure the internal mirror.
   

   # raidctl -c clt0d0 clt1d0

   -c clt0d0 clt1d0

   Creates the mirror of primary disk to the mirror disk. Enter the name of your primary disk as the first argument. Enter the name of the mirror disk as the second argument.

3. Continue with installing and configuring your multipathing software, if necessary, as instructed in the Sun Cluster installation guide.
   
4. Install the Sun Cluster software, as instructed in the Sun Cluster installation guide.
   

How to Configure Internal Disk Mirroring After the Cluster is Established

Before You Begin

This procedure assumes that you have already installed your hardware and software and have established the cluster.

Check the PatchPro site for any patches required for using internal disk mirroring.

PatchPro is a patch-management tool that eases the selection and download of patches required for installation or maintenance of Sun Cluster software. PatchPro provides an Interactive Mode tool especially for Sun Cluster. The Interactive Tool makes the installation of patches easier. PatchPro's Expert Mode tool helps you to maintain your configuration with the latest set of patches. Expert Mode is especially useful for obtaining all of the latest patches, not just the high availability and security patches.

To access the PatchPro tool for Sun Cluster software, go to http://www.sun.com/PatchPro/, click Sun Cluster, then choose either Interactive Mode or Expert Mode. Follow the instructions in the PatchPro tool to describe your cluster configuration and download the patches.

For third-party firmware patches, see the SunSolve™ Online site at http://sunsolve.ebay.sun.com.

1. If necessary, prepare the node for establishing the mirror.
   
   1. Determine the resource groups and device groups that are running on the node.
      
      Record this information because you use this information in later in this procedure to return resource groups and device groups to the node.
      

      # scstat

   2. If necessary, move all resource groups and device groups off the node.
      

      # scswitch -S -h fromnode

2. Configure the internal mirror.
   

   # raidctl -c clt0d0 clt1d0

   -c clt0d0 clt1d0

   Creates the mirror of primary disk to the mirror disk. Enter the name of your primary disk as the first argument. Enter the name of the mirror disk as the second argument.

3. Boot the node into single user mode.
   

   # reboot –– -S

4. Clean up the device IDs.
   

   # scdidadm -R /dev/rdsk/clt0d0

   -R /dev/rdsk/clt0d0

   Updates the cluster's record of the device IDs for the primary disk. Enter the name of your primary disk as the argument.

5. Confirm that the mirror has been created and only the primary disk is visible to the cluster.
   

   # scdidadm -l

   
   The command lists only the primary disk as visible to the cluster.
   

6. Boot the node back into cluster mode.
   

   # reboot

7. If you are using Solstice DiskSuite or Solaris Volume Manager and if the state database replicas are on the primary disk, recreate the state database replicas.
   

   # metadb -afc 3 /dev/rdsk/clt0d0s4

8. If you moved device groups off the node in Step 1, move all device groups back to the node.
   
   Perform the following step for each device group you want to return to the original node.
   

   # scswitch -z -D devicegroup -h nodename

   
   In this command, devicegroup is one or more device groups that are returned to the node.
   

9. If you moved resource groups off the node in Step 1, move all resource groups back to the node.
   

   # scswitch -z -g resourcegroup -h nodename

How to Remove an Internal Disk Mirror

1. If necessary, prepare the node for removing the mirror.
   
   1. Determine the resource groups and device groups that are running on the node.
      
      Record this information because you use this information later in this procedure to return resource groups and device groups to the node.
      

      # scstat

   2. If necessary, move all resource groups and device groups off the node.
      

      # scswitch -S -h fromnode

2. Remove the internal mirror.
   

   # raidctl -d clt0d0

   -d clt0d0

   Deletes the mirror of primary disk to the mirror disk. Enter the name of your primary disk as the argument.

3. Boot the node into single user mode.
   

   # reboot –– -S

4. Clean up the device IDs.
   

   # scdidadm -R /dev/rdsk/clt0d0
   # scdidadm -R /dev/rdsk/clt1d0

   -R /dev/rdsk/clt0d0

   -R /dev/rdsk/clt1d0

   Updates the cluster's record of the device IDs. Enter the names of your disks separated by spaces.

5. Confirm that the mirror has been deleted and that both disks are visible.
   

   # scdidadm -l

   
   The command lists both disks as visible to the cluster.
   

6. Boot the node back into cluster mode.
   

   # reboot

7. If you are using Solstice DiskSuite or Solaris Volume Manager and if the state database replicas are on the primary disk, recreate the state database replicas.
   

   # metadb -c 3 -ag /dev/rdsk/clt0d0s4

8. If you moved device groups off the node in Step 1, return the device groups to the original node.
   

   # scswitch -z -D devicegroup -h nodename

9. If you moved resource groups off the node in Step 1, return the resource groups and device groups to the original node.
   
   • If you are using Sun Cluster 3.2, use the following command:
     
     Perform the following step for each resource group you want to return to the original node.
     

     # clrg switch -h nodename resourcegroup

   • If you are using Sun Cluster 3.1, use the following command:
     
     Perform the following step for each resource group you want to return to the original node.
     

     # scswitch -z -g resourcegroup -h nodename

Top

SPARC: Support for Sun StorEdge QFS With Oracle 10g Real Application Clusters

Sun Cluster Support for Oracle Real Application Clusters supports the use of Sun StorEdge QFS with Oracle 10g Real Application Clusters. For information about Sun Cluster Support for Oracle Real Application Clusters, see Sun Cluster Data Service for Oracle Real Application Clusters Guide for Solaris OS.

Oracle 10g Real Application Clusters introduces new types of files. For information about using Sun StorEdge QFS for these new types of files, see the subsections that follow.

Top

SPARC: Requirements for Using the Sun StorEdge QFS Shared File System

You can store all of the files that are associated with Oracle Real Application Clusters on the Sun StorEdge QFS shared file system.

For information about how to create a Sun StorEdge QFS shared file system, see the following documentation for Sun StorEdge QFS:

• Sun StorEdge QFS and Sun StorEdge SAM-FS Software Installation and Configuration Guide
• Sun StorEdge QFS and Sun StorEdge SAM-FS File System Administration Guide

Distribute these files among several file systems as explained in the subsections that follow.

Top

Sun StorEdge QFS File Systems for RDBMS Binary Files and Related Files

For RDBMS binary files and related files, create one file system in the cluster to store the files.

The RDBMS binary files and related files are as follows:

• Oracle relational database management system (RDBMS) binary files
• Oracle configuration files (for example, init.ora, tnsnames.ora, listener.ora, and sqlnet.ora)
• System parameter file (SPFILE)
• Alert files (for example, alert_sid.log)
• Trace files (*.trc)
• Oracle Cluster Ready Services (CRS) binary files

Top

Sun StorEdge QFS File Systems for Database Files and Related Files

For database files and related files, determine whether you require one file system for each database or multiple file systems for each database.

• For simplicity of configuration and maintenance, create one file system to store these files for all Oracle Real Application Clusters instances of the database.
• To facilitate future expansion, create multiple file systems to store these files for all Oracle Real Application Clusters instances of the database.

  Note If you are adding storage for an existing database, you must create additional file systems for the storage that you are adding. In this situation, distribute the database files and related files among the file systems that you will use for the database.

Each file system that you create for database files and related files must have its own metadata server. For information about the resources that are required for the metadata servers, see Resources for the Sun StorEdge QFS Shared File System.

The database files and related files are as follows:

• Data files
• Control files
• Online redo log files
• Archived redo log files
• Flashback log files
• Recovery files
• Oracle cluster registry (OCR) files
• Oracle CRS voting disk

Top

SPARC: Resources for the Sun StorEdge QFS Shared File System

If you are using the Sun StorEdge QFS shared file system, answer the following questions:

• Which resources will you create to represent the metadata server for the Sun StorEdge QFS shared file system?
  
  One resource is required for each Sun StorEdge QFS metadata server.
• Which resource groups will you use for these resources?
  
  You might use multiple file systems for database files and related files. For more information, see Requirements for Using the Sun StorEdge QFS Shared File System.
  
  If you are using Oracle 10g, Oracle CRS manage Real Application Clusters database instances. These database instances must be started only after all shared file systems are mounted. To meet this requirement, ensure that the file system that contains the Oracle CRS voting disk is mounted only after the file systems for other database files have been mounted. This behavior ensures that, when a node is booted, Oracle CRS are started only after all Sun StorEdge QFS file systems are mounted.
  
  To enable Sun Cluster to mount the file systems in the required order, configure resource groups for the metadata servers of the file systems as follows:
  • Create the resources for the metadata servers in separate resource groups.
  • Set the resource group for the file system that contains the Oracle CRS voting disk to depend on the other resource groups.

For more information, see the following documentation for Sun StorEdge QFS:

• Sun StorEdge QFS and Sun StorEdge SAM-FS Software Installation and Configuration Guide
• Sun StorEdge QFS and Sun StorEdge SAM-FS File System Administration Guide

Use the answers to these questions when you perform the procedure in Registering and Configuring Oracle RAC Server Resources in Sun Cluster Data Service for Oracle Real Application Clusters Guide for Solaris OS.

Top

Support for Sun Cluster HA for Oracle on the x86 Platform

Sun Cluster supports the use of the Sun Cluster HA for Oracle data service on the x86 platform. For use with Sun Cluster HA for Oracle on the x86 platform, Oracle 10g is required.

For information about how to configure Sun Cluster HA for Oracle on the x86 platform, see Sun Cluster Data Service for Oracle Guide for Solaris OS.

In some situations, unnecessary error messages about the unavailability of the UNIX Distributed Lock Manager (Oracle UDLM) might be displayed. To prevent these error messages from being displayed, prevent the Oracle cssd daemon from being started. For instructions, see Support for Oracle 10g.

Top

Support for Automatic Storage Management (ASM) With Oracle 10g Real Application Clusters on the SPARC Platform

Sun Cluster supports the use of ASM with Oracle 10g Real Application Clusters on the SPARC platform.

Top

Required Versions of Oracle 10g Real Application Clusters

If you are using ASM, you must use Oracle 10g Real Application Clusters version 10.1.0.3 with the following Oracle patches:

• 3644481
• 3976437

How to Use ASM with Oracle 10g Real Application Clusters

Except as indicated in this section, the procedures for using ASM with Sun Cluster Support for Oracle Real Application Clusters are identical to the procedures for using hardware redundant array of independent disks (RAID). For more information about these procedures, see Sun Cluster Data Service for Oracle Real Application Clusters Guide for Solaris OS.

1. Run the scdidadm(1M) command to find the raw device identity (DID) that corresponds to shared disks that are available in the cluster.
   
   The following example lists output from the scdidadm -L command.
   

   # scdidadm -L
   1 phys-schost-1:/dev/rdsk/c0t2d0 /dev/did/rdsk/d1
   1 phys-schost-2:/dev/rdsk/c0t2d0 /dev/did/rdsk/d1
   2 phys-schost-1:/dev/rdsk/c0t3d0 /dev/did/rdsk/d2
   2 phys-schost-2:/dev/rdsk/c0t3d0 /dev/did/rdsk/d2

2. Use the DID that the scdidadm output identifies to set up the disk in the ASM disk group.
   
   For example, the scdidadm output might identify that the raw DID that corresponds to the disk is d2. In this instance, use the /dev/did/rdsk/d2sN raw device, where N is the slice number.
   

Top

Restrictions and Requirements

The following restrictions and requirements have been added or updated since the Sun Cluster 3.1 9/04 release.

Top

Restriction on SCI Card Placement

Do not place an SCI card in the 33 MHz PCI slot (slot 1) of the hot swap PCI+ (hsPCI+) I/O assembly. This placement can cause a system panic.

Top

Storage-Based Data Replication and Quorum Devices

When using storage-based data replication, with storage devices that provide this feature, never configure a replicated volume as a quorum device.

Top

Known Problems

In addition to known problems that are documented in Sun Cluster 3.1 9/04 Release Notes for Solaris OS, the following known problems affect the operation of the Sun Cluster 3.1 9/04 release.

Top

Bug ID 4333241

Problem Summary: System deadlocks when using jumbo frames with Solaris 8 and failover or scalable data services.

Workaround: Support of jumbo frames with Solaris 8 is limited to clusters using Oracle Real Application Clusters only. Solaris 9 can be used with all types of data services

Top

Bug ID 4804696

Problem Summary: If an attempt is made by VxVM to re-encapsulate root on a device that was previously encapsulated, an error can result due to not being able to create the rootdg:

Workaround: Using the touch command, create an empty install-db file in the /etc/vx/reconfig.d/state.d directory. Then kill the vxconfigd daemon and recreate the daemon in disable mode.

After performing these steps, you should be able to re-encapsulate root.

Top

Bug ID 5107076

Problem Summary: When Sun Cluster software is upgraded from a previous version to Sun Cluster 3.1 9/04 release, the Sun Cluster support packages for Oracle Real Application Clusters are not upgraded.

Workaround: When you upgrade Sun Cluster software to Sun Cluster 3.1 9/04 release, you must remove the Sun Cluster support packages for Oracle Real Application Clusters from the Sun Cluster system and add the Sun Cluster support packages from the Sun Java Enterprise System Accessory CD Volume 3.

How to Replace the Sun Cluster Support Packages for Oracle Real Application Clusters

1. Load the Sun Java Enterprise System Accessory CD Volume 3 into the CD-ROM drive.
   
2. Become superuser.
   
3. Change the current working directory to the directory that contains the packages for the Real Application Clusters framework resource group.
   
   This directory depends on the version of the Solaris Operating System that you are using.
   
   • If you are using Solaris 8, run the following command:

     # cd /cdrom/cdrom0/components/SunCluster_Oracle_RAC/Solaris_8/Packages

   • If you are using Solaris 9, run the following command:

     # cd /cdrom/cdrom0/components/SunCluster_Oracle_RAC/Solaris_9/Packages

4. On each cluster node that can run Sun Cluster Support for Oracle Real Application Clusters, transfer the contents of the required software packages from the CD-ROM to the node.
   
   The required software packages depend on the storage management scheme that you are using for the Oracle Real Application Clusters database.
   
   • If you are using Solaris Volume Manager for Sun Cluster, run the following commands:

     # pkgrm SUNWudlm SUNWudlmr SUNWschwr SUNWscucm
     # pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWscmd

   • If you are using VxVM with the cluster feature, run the following commands:

     # pkgrm SUNWudlm SUNWudlmr SUNWcvmr SUNWcvm SUNWscucm
     # pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWcvmr SUNWcvm

   • If you are using hardware RAID support, run the following commands:

     # pkgrm SUNWudlm SUNWudlmr SUNWschwr SUNWscucm
     # pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWschwr

   • If you are using Sun StorEdge QFS shared file system with hardware RAID support, run the following commands:

     # pkgrm SUNWudlm SUNWudlmr SUNWschwr SUNWscucm
     # pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWschwr

Top

Bug ID 5109935

Problem Summary: When using scinstall to upgrade Sun Cluster data services for the Sun Cluster 3.1 9/04 release, Sun Cluster will issue error messages complaining about missing Solaris_10 Packages directories.

Workaround: These error messages can be safely ignored.

Top

Bug ID 6196936

Problem Summary: SCSI reset errors when using X4422A Sun Dual Gigabit Ethernet + Dual SCSI PCI Adapter cards in Sun Fire V40z's PCI slots 2 and 3.

Workaround: Do not use 4422A cards in both slot 2 and 3.

Top

Bug ID 6198608

Problem Summary: An underlying firmware problem caused by issuing an SCMD_READ_DEFECT_LIST (0x37) to an SE 3510 disk causes clusters to panic when run with Explorer versions 4.3 or 4.3.1 (these versions call diskinfo -g). The Sun Cluster sccheck command in Sun Cluster 3.1 (10/03) through Sun Cluster 3.1 (9/04) allows Explorer to run the command that causes the panic. Java Enterprise System R3 also includes Explorer 4.3.1. This scsi command can be issued by either using format (defect->grown option) or by running Explorer 4.3 and 4.3.1.

Workaround: Release 4.1 of the SE 3510 firmware contains the fix to the problem. Sun Cluster 3.1 (5/05) will include a workaround to the problem when it occurs by using sccheck. There is also a workaround for the problem in Explorer 4.4. EMC Clarion arrays have also experienced this problem. Contact EMC to obtain the appropriate firmware fix.

Top

Bug ID 6210418

Problem Summary: If a process accounting log is located on a cluster file system or on an HAStoragePlus failover file system, a switchover would be blocked by writes to the log file. This would then cause the node to hang.

Workaround: Use only a local file system to contain process accounting log files.

Top

Bug ID 6220218

Problem Summary: The standard license for Veritas Storage Foundation 4.0 is enabling the VxVM Persistent Group Reservations (PGR) functionality, making the product incompatible with Sun Cluster software. This incompatibility might bring down the cluster by causing the cluster nodes to panic.

Workaround: Download from http://www.sunsolve.com Patch 120585 (revision -01 or higher) and follow the Special Install Instructions at the end of the patch description to apply the patch to your cluster.

Top

Bug ID 6252555

Problem Summary: The sd driver patches 113277-28 and higher break quorum reservations, resulting in a node panic.

Workaround: Do not use patch 113277-28 or later, until further notice, if the target cluster uses one of the following arrays as shared storage:

• Sun StorEdge 3510
• Sun StorEdge 3511
• Sun StorEdge 6120
• Sun StorEdge 6130
• Sun StorEdge 6920

and if one or more volumes within the array is visible to more than 2 nodes of a Sun Cluster 3 cluster. Sun Alert 101805 provides more information about this issue.

Top

Known Documentation Problems

This section discusses documentation errors you might encounter and steps to correct these problems. This information is in addition to known documentation problems documented in the Sun Cluster 3.1 9/04 Release Notes for Solaris OS.

Top

System Administration Guide

The following subsection describes omissions or new information that will be added to the next publication of the Sun Cluster System Administration Guide for Solaris OSSystem Administration Guide.

Top

Cluster File System Restrictions

In the section Cluster File System Restrictions, the list of restrictions is not correct. The Sun Cluster 3.1 9/04 release supports the -f option of the umount command. The Sun Cluster 3.1 9/04 release supports forced unmounts.

Top

Software Installation Guide

The following subsections describe omissions or new information that will be added to the next publication of the Sun Cluster Software Installation Guide for Solaris OS.

Top

IPv6 Support and Restrictions for Public Networks

Sun Cluster software supports IPv6 addresses on the public network under the following conditions or restrictions:

• Sun Cluster software does not support IPv6 addresses on the public network if the private interconnect uses SCI adapters.
• On Solaris 9 OS, Sun Cluster software supports IPv6 addresses for both failover and scalable data services.
• On Solaris 8 OS, Sun Cluster software supports IPv6 addresses for failover data services only.

Top

IPv6 Requirement for the Cluster Interconnect

To support IPv6 addresses on the public network, Sun Cluster software requires that all private network adapters must use network interface cards (NICs) that support local MAC address assignment. Link-local IPv6 addresses, which are required on private network addresses to support IPv6 public network addresses, are derived from the local MAC addresses.

Top

Correction to the Upgrade of Sun Cluster HA for SAP liveCache

The procedure How to Finish a Nonrolling Upgrade to Sun Cluster 3.1 9/04 Software includes an instruction to modify the /opt/SUNWsclc/livecache/bin/lccluster file. This instruction applies if you upgraded the Sun Cluster HA for SAP liveCache data service from the Sun Cluster 3.0 version to the Sun Cluster 3.1 version. This instruction is incorrect.

Do not perform Step 3, the instruction to edit the /opt/SUNWsclc/livecache/bin/lccluster file. This file is only a template that is installed with the Sun Cluster HA for SAP liveCache data service. Do not edit the lccluster file at that location. Instead, perform the following procedure:

How to Upgrade Sun Cluster HA for SAP liveCache to Sun Cluster 3.1

1. Go to a node that will host the Sun Cluster HA for SAP liveCache resource.
   
2. Copy the new /opt/SUNWsclc/livecache/bin/lccluster file to the /sapdb/LC-NAME/db/sap/ directory.
   
   Overwrite the lccluster file that already exists from the previous configuration of the data service.
   
3. Configure this /sapdb/LC-NAME/db/sap/lccluster file as documented in Step 3 of How to Register and Configure Sun Cluster HA for SAP liveCache in Sun Cluster Data Service for SAP liveCache Guide for Solaris OS.
   

Top

Man Pages

The following subsections describe omissions or new information that will be added to the next publication of the man pages.

Top

Data-Service Names for Individual Upgrade

The following table lists the names to specify to the scinstall -u update -s srvc command. A version of the data service must already be installed for the command to succeed.

scinstall Upgrade Names of Data Services