Installing Solaris 10 5/08 OS on the IBM LS42 Blade Server

by Lee Damico

Description

In order to install the IBM LS42 blade server with the Solaris 10 5/08 OS, it is necessary to create a custom install image and perform some post install steps.

But first, you should verify that the Advanced Management Module (AMM) firmware on the BladeCenter is Version 1.42N Build ID: BPET42N (or later). This update is required in order to avoid problems with keyboard and mouse communications between the AMM KVM and the blade. Without this update, the LS42 blade server may hang under certain conditions. For details see document ID MIGR-5076204 on the IBM website or visit the weblink:

https://www-304.ibm.com/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-5076204&brandind=5000020

Problems relating to Solaris 10 5/08 on the IBM LS42 blade server include the following:

CR 6713128 Maximum payload size on some slots set incorrectly

This problem is exhibited in the form of warning messages from the mpt driver
and the system appearing to hang. You will see the following warning messages:

WARNING: /pci@0,0/pci1166,140@7/pci10b5,8518@0/pci10b5,8518@1/
pci1000,1000@0 (mpt0):
Disconnected command timeout for Target 0

This is addressed by patch ID 137112-04. This patch is available from SunSolve at:

http://sunsolve.sun.com/search/document.do?assetkey=1-21-137112-04-1

CR 6637053 The bnx driver should support 5709 chip

Because the Solaris 10 5/08 bnx driver does not yet support the onboard Broadcom NetXtreme II 5709s NIC, reliance on this as the default NIC requires installation of the latest version of the Broadcom bnx driver available for download direct from Broadcom:

http://www.broadcom.com/support/ethernet_nic/netxtremeii.php

Choose the Solaris GLDv3 4.4.3 (or later) version.

CR 6510830 SPARC and x86 PCIe IO error handling should be merged (bnx)

This problem is exhibited by the Broadcom bnx driver v4.4.3. It results in netinstall hanging as well as regular boot hanging on an intermittent basis.

This problem is also addressed by installation of patch ID 137112-04. In addition, an update to /etc/system is required to set pci_error_disable_flag.

CR 6740922 Loads of cpudrv: WARNING: notices

You may see the following warning messages:

WARNING: cpudrv_power: cpudrv_pm_change_speed() return = -1
WARNING: Device /cpus/cpu@3 failed to power up.
WARNING: cpudrv_pm_monitor: instance 3: can't raise CPU power level

These messages are harmless and can be ignored.

Workaround

The workaround for installing the LS42 BladeCenter involves creating a custom Solaris 10 5/08 install image that incorporates patch ID 137112-04 as well as the Broadcom BRCMbnx package. In addition, there are a few pre-install and post-install steps required. These procedures are documented below.

Creating a Custom Solaris 10 5/08 Install Image

The following procedures are specific to creating a modified Solaris 10 5/08 install image. For information on creating a network install server, see the Solaris 10 5/08 Release and Installation Collection, Creating an Install Server with DVD Media:

http://docs.sun.com/app/docs/doc/820-4040/netinstall-94672?a=view

Download patch ID 137112-04 from the SunSolve site and at least version 4.4.3 of the GLDv3 bnx driver from the Broadcom site. Extract the contents of both of these downloaded files. You should have a 137112-04 patch directory and a BRCMbnx package directory.

The following naming conventions are used for creating the custom install image:

1. Become superuser or assume an equivalent role.
2. Insert the Solaris DVD into the system's drive.
3. Create a directory to contain the boot image.

   # mkdir -p install_dir_path
   

4. Change to the Tools directory on the mounted disc:

   # cd /cdrom/cdrom0/Solaris_10/Tools
   

5. Copy the disc in the drive to the install server's hard disk by using the setup_install_server command:

   # ./setup_install_server install_dir_path
   

6. Unpack the network installation boot archive.

   # /boot/solaris/bin/root_archive unpackmedia install_dir_path \
           destination_dir
   

   This will create the directory destination_dir which should now contain the
   extracted boot archive contents.

7. Patch the unpacked boot archive with the Sun patch 137112-04 and the Broadcom BRCMbnx package.
   NOTE: The BRCMbnx package must first be removed from the miniroot with pkgrm in order for the new BRCMbnx package to be added successfully. Otherwise, the /etc/driver_aliases file will not be properly updated.

   # patchadd -d -C destination_dir path_to_patch/137112-04
   # pkgrm -R destination_dir BRCMbnx
   # pkgadd -d path_to_patch -R destination_dir BRCMbnx
   

8. Update the Patches directory of the Solaris 10 5/08 image to include patch 137112-04 and update the Product directory to replace the Sun BRCMbnx package with the Broadcom BRCMbnx package. Doing so will eliminate the need to do a manual post install of the patch and package. See Step 3 under Using the Custom Image to Install the LS42 Blade Server for more information.

   # cd install_dir_path/Solaris_10/Patches
   # cp -r path_to_patch/137112-04 .
   # cd install_dir_path/Solaris_10/Product
   # rm -rf BRCMbnx
   # cp -r path_to_patch/BRCMbnx .
   

9. Update /etc/system in the boot archive to add the following line at the end of the file:

   set pcie:pcie_error_disable_flag=1
   

   # cd destination_dir/etc
   # vi system
   

   Note: You will need to do this again during postinstall. If you forget to do this either in the boot archive or during the post install procedures, you will need to boot the kernel in kmdb and enter the following at the kmdb prompt [0]>:

   ::bp pcie`_init
   :c
   pcie_error_disable_flag/W 1
   :c
   

10. Pack the x86 boot archive.

    # /boot/solaris/bin/root_archive packmedia install_dir_path \
            destination_dir
    

11. If destination_dir – the extracted boot archive – was created within the Solaris 10 5/08 image, after repacking the archive, this temporary directory should be manually removed.

    # rm -rf destination_dir
    

12. If you wish to create a DVD media from this image, do the following:

    # mkisofs -o new_install_image -b boot/grub/stage2_eltorito \
    -c .catalog -no-emul-boot -boot-load-size 4 -boot-info-table \
    -relaxed-filenames -N -allow-leading-dots -l -r -J -d -D \
    -V SOL_10_U5MOD install_dir_path
    

    where new_install_image will be the newly created iso file containing the iso image in install_dir_path.

Using the Custom Image to Install the LS42 Blade Server

Now that you have a custom image, there are still a few things you need to do in order to successfully install the system.

1. Set the boot order via <F12> to default to the method of installation, for example, Network or CD.
2. If you forgot to update the boot archive /etc/system file as per Step 9 above, on display of the GNU Grub menu, make sure you edit the boot flags to include -kd. This will boot the kernel in kmdb mode. Once in kmdb mode, follow the instructions in Step 9.
3. When the Solaris Installation menu is displayed, choose option 4. Solaris Interactive Text (console session). Doing so will result in the patch you placed in the custom install image Patches directory being automatically installed during post install.
4. As you progress through the install options, you will be asked whether or not to Reboot After Installation? Choose Manual Reboot. This will allow you to update /etc/system as per Step 9 so that you do not have to boot the kernel in kmdb mode each time you boot.
5. After the install completes, assuming you specified Manual Reboot, you will be instructed to press <Return> to reboot or ! to exit to the shell. Press ! to exit to the shell and then edit /a/etc/system to include the following line:

   set pcie:pcie_error_disable_flag=1
   

6. Reboot the system