Section 3 - Redundancy

{section}
{column:width=25%}
h5. [LDoms Community Cookbook]\\
h6. Contents
{pagetree:root=@parent|searchBox=true}
h6. In this Section ...
{toc:type=list|style=none|maxLevel=3}
{column}
{column:width=75%}

h1. Section Introduction

h6. (Contributed by Narayan Venkat)
h2. Redundant Service Domains
*Configure the primary service domain*
A system configured for virtual IO failover, consists of two (primary & alternate) service domains with direct IO access, and client domain(s) configured to be serviced by both service domains. In order to create a two service domains with direct access to physical IO, the PCI-E bus in the system has to be split and assigned to both service domains. In this example, on a T2000, we will start with the "factory-default" configuration, and create and configure the "primary" domain such that it only uses one PCI-E bus (bus_a). To do this configuration on a different platform, the primary domain would have to be configured with the appropriate PCI bus depending on the system. The "primary" service domain like a typical service domain will also run a virtual disk server and virtual network switch. The VIO services will use the physical disk and network adapter available on PCI-E bus_a.

{panel:borderStyle=solid| borderColor=black|bgColor=white}
!redundant_vio_topology_v0.1.png!
h6. Redundant virtual I/O services topology
{panel}

{note:Title=Boot/Network Devices on both Busses}
Before removing bus_b from the primary domain, ensure that both bus_a and bus_b contain disk and network devices for use by the respective service domains. For more information on determining which bus to remove, review [Section 1 - Setup, Hardware & Split-PCI|http://wikis.sun.com/pages/viewpage.action?pageId=51678341]
{note}

* Specify CPU amd Memory resources
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm set-vcpu 4 primary*}}
{{primary# *ldm set-mem 4G primary*}}
{panel}

* Remove bus_b from Primary domain
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm rm-io bus_b primary*}}
{panel}

* Add vds, vsw, vcc services
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vds primary-vds0 primary*}}
{{primary# *ldm add-vsw net-dev=e1000g0 primary-vsw0 primary*}}
{{primary# *ldm add-vcc port-range=5000-5100 primary-vcc0 primary*}}
{panel}

* Save the new configuration to the SP and reboot the system:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-spconfig initial*}}
{{primary# *init 6*}}
{panel}

*Configure the alternate service domain*
Create and configure the alternate service domain just like any other domain. Instead of creating virtual disk and network devices, the domain is also configured like the primary domain with disk server and virtual switch services that use the physical devices available on its PCI-E bus_b.

* Create alternate service domain:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm create alternate*}}
{panel}

* Specify CPU amd Memory resources:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm set-vcpu 4 alternate*}}
{{primary# *ldm set-mem 4G alternate*}}
{panel}

* Add bus_b to the alternate domain:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-io bus_b alternate*}}
{panel}

* Add vds, vsw services to the alternate service domain:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vds alternate-vds0 alternate*}}
{{primary# *ldm add-vsw net-dev=e1000g3 alternate-vsw0 alternate*}}
{panel}

* Add vds, vsw services to the alternate service domain:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm bind alternate*}}
{{primary# *ldm start alternate*}}
{panel}

* Connect to the alternate service domain console:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *telnet localhost 5000*}} <-- your port number may vary!
{panel}

*Create the client ldg1 domain*
We now create a domain called ldg1. This domain will be connected to both the control and alternate service domains for disk and network services.

* Create domain ldg1 and specify VCPU and memory:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm create ldg1*}}
{{primary# *ldm set-vcpu 4 ldg1*}}
{{primary# *ldm set-mem 4G ldg1*}}
{panel}

* Export two files as virtual disks from both virtual disk servers. The two files should have the same size as there are going to be used as mirrors. If you have an existing bootdisk image then export that image from the primary domain and export a blank file from the alternate domain. The file from the alternate will be update when the mirror is created:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /primary/ldg1/bootdisk.img vol1@primary-vds0*}}
{{primary# *ldm add-vdsdev /alternate/ldg1/bootdisk.img vol1@alternate-vds0*}}
{panel}

* Add two virtual disks, each one is connected to a different service domain. Each vdisk should be configured with a timeout (10 seconds for example) so that the vdisk will return an error if it cannot establish a connection with the vdisk server after the specified period, and this will cause the mirror to only use the sub-mirror which is accessible:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk timeout=10 vdisk1 vol1@primary-vds0 ldg1*}}
{{primary# *ldm add-vdisk timeout=10 vdisk2 vol1@alternate-vds0 ldg1*}}
{panel}

* Add two vnet devices,each one is connected to a different service domain:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vnet vnet1 primary-vsw0@primary ldg1*}}
{{primary# *ldm add-vnet vnet2 alternate-vsw0@alternate ldg1*}}
{panel}

* Configure both disks as boot device, and set auto-boot? to true:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm set-var boot-device="/virtual-devices@100/channel-devices@200/disk@0/virtual-devices@100/channel-devices@200/disk@1" ldg1*}}
{{primary# *ldm set-var auto-boot\?=true ldg1*}}
{panel}

* Bind and start the guest domain (ldg1):
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm bind ldg1*}}
{{primary# *ldm start ldg1*}}
{panel}

* Connect to the guest domain console:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *telnet localhost 5001*}} <-- your port number may vary!
{panel}

h2. Boot Disk Mirroring with SVM
The client domain consists of two virtual disks backed (c0d0 and c0d1) by files exported by the virtual disk server (these could also be provided by redundant service domains).

* Check the partitioning of the bootdisk. Here the system was installed with slice 0 as the root (/) partition, slice 1 as swap. Slice 3 is a free slice that we are going to use for the SVM metadb:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *prtvtoc /dev/dsk/c0d0s2*}}
{{* /dev/dsk/c0d0s2 partition map}}
{{* .....}}
{{* .....}}
{{* .....}}
{{* First Sector Last}}
{{* Partition Tag Flags Sector Count Sector Mount Directory}}
{{0 2 00 0 14336400 14336399}}
{{1 3 01 14336400 1051200 15387599}}
{{2 5 00 0 16776000 16775999}}
{{3 0 00 15387600 1386600 16774199*}}
{panel}

* Configure the second vdisk so that it has the same partitioning as the first one:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *prtvtoc /dev/rdsk/c0d0s2 | fmthard -s - /dev/rdsk/c0d1s2*}}
{panel}

* Create the SVM metadb and metadevices:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *metadb -a -f -c3 c0d0s3 c0d1s3*}}
{{ldg1# *metainit -f d10 1 1 c0d0s0}}
{{ldg1# *metainit -f d11 1 1 c0d0s1}}
{{ldg1# *metainit -f d20 1 1 c0d1s0}}
{{ldg1# *metainit -f d21 1 1 c0d1s1}}
{panel}

* Create the SVM mirrors:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *metainit d0 -m d10*}}
{{ldg1# *metainit d1 -m d11*}}
{panel}

* Setup the root device. Save the current /etc/vfstab and /etc/system files so that can be used to restore the system to a pre-SVM configuration if necessary.
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *cp /etc/vfstab /etc/vfstab.saved*}}
{{ldg1# *cp /etc/system /etc/system.saved*}}
{{ldg1# *metaroot d0*}}
{panel}

{note:Ttile=Optional Disk Failure Protection}
Optional: make sure we can boot in case of disk failure. *USE ONLY WHERE DATA INTEGRITY IS LESS IMPORTANT THEN SERVICE AVAILABILITY. THIS CAN LEAD TO FILE CORRUPTION*. To make sure we can boot in case a disk fails, we need to tell the kernel to ignore the quorum on metadbs, otherwise we can't boot in a two disk setup, because we can't never fullfill the requirments. For that we add the following to /etc/system:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *echo "set md:mirrored_root_flag=1" >> /etc/system*}}
{panel}
{note}

* Reboot the domain. Following reboot, the system will come up for the first time on the mirror. If all went well we are up and running in a few seconds:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *reboot -- -r*}}
{panel}

* Add the disk mirrors. This will take considerable amount of time as the mirrors need to be synchronized. Use metastat to check on the progress of the syncing:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *metattach d0 d20*}}
{{ldg1# *metattach d1 d21*}}
{panel}

* Run metastat to ensure that that all the required mirroring is setup properly:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *metastat*}}
{{d1: Mirror}}
{{Submirror 0: d11}}
{{State: Okay}}
{{Submirror 1: d21}}
{{State: Okay}}
{{...}}
{{d0: Mirror}}
{{Submirror 0: d10}}
{{State: Okay}}
{{Submirror 1: d20}}
{{State: Okay}}
{{...}}
{panel}

{note:Title=Mirror Resync}
Since the metadisk in the client domain is mirrored, an explicit resync of the sub-mirrors is required following each service domain reboot. Repair the root and swap meta devices by adding the sub-mirrors:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *metareplace -e d0 c0d0s0*}}
{{ldg1# *metareplace -e d1 c0d0s1*}}
{panel}

The client domain ldg1 is now resilient to the reboot of either the primary or the alternate service domain. Disk access in ldg1 should failover on a service domain reboot.

h2. IP Failover with IPMP
The client domain consists of two virtual network devices. We will configure IP multipathing across these two devices. Obtain 4 IP addresses in the same subnet. In Multipathing there are 2 fixed (or private) address and 2 floating (or public) addresses. The 2 fixed addresses are referred to as internal. One is assigned directly to each network interface. The 2 floating addresses are the external ones. If one of the NICs detects link failure, the address tied to that NIC fails over to the working NIC. When the NIC comes back up, the address fails back to its original home. Determine right now which will be your internal IPs and which will be your external.

* Edit /etc/hosts with your 4 IPs. example:
{code:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
10.6.90.10 ldg1-vnet0
10.6.90.11 ldg1-vnet1
10.6.90.100 ldg1-ext0 ldg1-dummy
10.6.90.101 ldg1-ext1 ldg1.eng.sun.com
{code}

* Setup hostname.* files. You can pretty much copy these two files as is and just modify them slightly to fit your naming conventions in the same way that you setup the /etc/hosts file above. /etc/hostname.vnet0:
{code:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
/etc/hostname.vnet0
ldg1-vnet0 netmask + broadcast + group production deprecated -failover up \
addif ldg1-ext0 netmask + broadcast + failover up

/etc/hostname.vnet1
ldg1-vnet1 netmask + broadcast + group production deprecated -failover up \
addif ldg1-ext1 netmask + broadcast + failover up
{code}

* Configure any additional routing or default gateways as appropriate. For the above IP addresses, the following router was added as the default gateway:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *route add default 10.6.90.1*}}
{panel}

* Make it active. This is the easy part. Copy and paste your /etc/hostname.vnet* files to ifconfig commands as below:
{code:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
/sbin/ifconfig vnet0 ldg1-vnet0 netmask + broadcast + \
group production deprecated -failover up \
addif ldg1-ext0 netmask + broadcast + failover up

/sbin/ifconfig vnet1 ldg1-vnet1 netmask + broadcast + \
group production deprecated -failover up \
addif ldg1-ext1 netmask + broadcast + failover up
{code}

The client domain ldg1 is now resilient to the reboot of either the primary or the alternate service domain. Network access in ldg1 should failover on a service domain reboot.
----
h1. Cluster
h2. Sun Cluster implementation
Sun Cluster can be installed in any domain (guest or I/O domain). Each domain running Sun Cluster will be a node of a cluster and services can be switched over to any node of the same cluster, like with regular systems. When using Sun Cluster in a guest domain then the shared cluster disks must be virtual disks backed by a physical SCSI disk or LUN.
----
h1. Multipathing disk options
h2. Multipathing Overview
Multipathing solutions like MPxIO, EMC PowerPath, Hitachi HDLM, or Veritas DMP can only be used in I/O domains with physical disks. They can not used in guest domains with virtual disks.

h2. Solaris I/O Multipathing (aka MPxIO or STMS)

h2. Veritas Dymanic Multipathing (VxDMP)
If you want to export a disk managed by VxDMP as a virtual disk then you need to install patch 138042-01 or later.

h2. EMC Powerpath
If you want to export a disk managed by EMC powerpath as a virtual disk then you need to:
* install patch 138042-01 or later
* create the file /platform/sun4v/kernel/drv/vds.conf containing:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{driver-type-list="emcp:disk";}}
{panel}

* stop and unbind guest domains
* run the command:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *update_drv -v vds*}}
{panel}

* Rebind and restart the guest domains

h2. Hitachi Dynamic Link Manager (HDLM)
If you want to export a disk managed by Hitachi HDML as a virtual disk then you need to:
* install patch 138042-01 or later
* create the file /platform/sun4v/kernel/drv/vds.conf containing:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{driver-type-list="dlmfdrv:disk";}}
{panel}

* stop and unbind guest domains
* run the command:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *update_drv -v vds*}}
{panel}

* Rebind and restart the guest domains
{column}
{section}