Installing Solaris 10 5/08 OS on IBM x3850 M2 Server

by Lee Damico

NOTE: Some of the issues addressed by this Tech Tip have been fixed in the Solaris 10 10/08 release available for download at http://www.sun.com/software/solaris/get.jsp. These include:

CR 6658543 MCA poller cyclic on uninitialized handles does not short circuit
CR 6651366 Solaris needs native LSI mega_sas driver

The workaround for installing the bnx driver for support of the on board Broadcom 5709 NIC as well as the workaround for starting the Xorg server, both referenced below, are still required for Solaris 10 10/08.

Description

On the IBM x3850 M2 server, attempts to install or boot the Solaris 10 5/08 OS will result in a kernel panic at boot time. This problem is documented in:

CR 6658543 MCA poller cyclic on unitialized handles does not short circuit

Prior to the display of the installation menu, messages similar to the following will occur followed by the kernel panic:

SunOS Release 5.10 Version Generic_127128-11 32-bit
Copyright 1983-2008 Sun Microsystems, Inc.  All rights reserved.
Use is subject to license terms.
NOTICE: cmi_hdl_create: chipid 15 coreid 0 strandid 0 handle already allocated!
WARNING: There will be no MCA support on chip 15 core 0 strand 0 (cmi_hdl_create returned NULL)

NOTICE: cmi_hdl_create: chipid 8 coreid 1 strandid 0 handle already allocated!
WARNING: There will be no MCA support on chip 8 core 1 strand 0 (cmi_hdl_create returned NULL)

...

panic[cpu0]/thread=fec1ec20: BAD TRAP: type=e (#pf Page fault) rp=fec351f0 addr=6e6
b636f occurred in module "unix" due to an illegal access to a user address

Workaround

In order to install the Solaris 10 5/08 release, the kernel must be booted in debug mode via the kmdb(1) command so that cmi_no_init can be set. Once the Solaris OS is installed, it must be booted this way again and then once logged on, the /etc/system file must be updated to make this change permanent. These steps are outlined below.

NOTE: Before you start, make sure that you have disabled the on-board Broadcom 5709 NIC via the system BIOS since a Solaris driver is not yet available for this NIC. For network connectivity, install a supported alternate NIC in an available PCIe slot. See http://www.sun.com/bigadmin/hcl/data/systems/details/3406.html for supported NIC adaptors as well as other important support information.

Edit GNU GRUB Menu

As the system boots the Solaris 10 5/08 release, the following GNU GRUB menu will be displayed with Solaris highlighted as the default OS:

GNU GRUB version 0.95 (616K lower / 4127168K upper memory)
+-------------------------------------------------------------------+
|Solaris                                                            |
|Solaris  failsafe                                                  |                                              |
+-------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted. Press
enter to boot the selected OS, 'e' to edit the commands before
booting, or 'c' for a command-line.

*Press e to edit
*Press e again to edit the boot line
*Append <SPACE> -kd to the end of the line and then press <RETURN>
*Press b to boot

Enter kmd Command

At the kmd prompt [0]>, enter the cmi_no_init/W1 command:

   [0]> cmi_no_init/W1

The system will display:

   cmi_no_init: 0       =       0x1

Enter :c to continue booting:

   [0]> :c

Note: During this process, if you are not quick enough, you may encounter the following message one or more times:

kmdb: Do you really want to reboot? (y/n)

Answer n until you get the [0]> prompt again. Then enter the cmi_no _init command and the :c command as documented above.

Apply Appropriate Install Time Updates (ITU)

After successfully doing the above, the system will boot and present you with the Solaris installation menu. Choose option 5. Apply driver updates in order to install the mega_sas driver, and if appropriate, the bnx driver, both described in Driver Info section on the IBM x3850 M2 HCL listing: http://www.sun.com/bigadmin/hcl/data/systems/details/3406.html.

These are further documented in:

6651366 Solaris needs native LSI mega_sas driver
6637053 bnx driver should support 5709c chip

Begin Installation Configuration

After installation of the necessary drivers, press e <RETURN> to go back to the install menu. Choose option 4. Solaris Interactive Text (console session). This is necessary due to a bug in Xorg and is documented at: http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/6668806_xorg_server

You will eventually be given the option of an automatic reboot or a manual reboot after completion of the install. Choose Manual Reboot. This will allow you to update the /etc/system file and set cmi_no_init so that it will be unnecessary to boot in kmdb mode on subsequent reboots.

Reboot the System and Update the /etc/system File

After installation completes, if you specified a Manual Reboot after installation, you can now exit to the shell (via !) and edit /a/etc/system to include the following line:

    set cmi_no_init=1

If you failed to set the system to Manual Reboot and were unable to update the /etc/system file before the system automatically rebooted, then the system will eventually panic on boot. At this point, you will need to reboot the system and perform the procedures documented in the Edit GNU GRUB Menu and Enter kmd Command procedures above in order to boot in kmdb mode. If the install media was not automatically ejected after completion of the installation, you will also need to do so at this time before rebooting the system.

When the system has booted and you have a login prompt, log in as root and add the following to /etc/system:

    set cmi_no_init=1

Once this is done, it will no longer be necessary on subsequent reboots to go through the kmdb boot procedure documented above.

Launching the Xorg server

The following bug results in a failure to start the Xorg server:

6668806 Xorg server fails to start on IBM x3850 M2

In order to successfully start the Xorg server, you will need to create a xorg.conf file as documented in http://www.sun.com/bigadmin/jsp/descFile.jsp?url=descAll/6668806_xorg_server.