{section}
{column:width=25%}
h5. [LDoms Community Cookbook]
h6. Contents
{pagetree:root=@parent|searchBox=true}
h6. In this Section ...
{toc:type=list|style=none|maxLevel=3}
{column}
{column:width=75%}
h1. Section Introduction
This section provides background information on the virtual devices and services in Logical Domains, and provides examples on how to configure and use them.
----
h1. Virtual Networks
h2. Service Domain to Guest Domain Traffic
If you have a service domain with a virtual switch (for example primary-vsw0 in the primary) and guest domains using that virtual switch then, by default, guest domains will not be able to communicate with the service domain over the network.
To enable network communications between guest domains and a service domain then you need to plumb the interface of the virtual switch. So what you will usually do is plumb the virtual switch interface instead of the physical network interface the virtual switch is associated with.
For example, if your virtual switch is associated with the e1000g0 interface in the primary domain:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm ls \-l primary{*}}}
{{...}}
{{VSW}}
{{NAME MAC NET-DEV DEVICE DEFAULT-VLAN-ID PVID VID MODE}}
{{primary-vsw0 00:14:4f:fb:54:f2 e1000g0 switch@0 1 1}}
{{...}}
{panel}
Then you are going to plumb the vsw0 interface instead of e1000g0. The vsw0 interface should be configured with the same parameters (IP address, netmask, broadcast...) as the e1000g0 interface while the e1000g0 is going to be unplumbed.
{note:title=Use the Console!}
Switching from the e1000g0 interface to the vsw0 interface should be done from the console (or from an alternate network connection) as the e1000g0 network connection will be brought down during the switch.
{note}
* plumb the vsw interface
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig vsw0 plumb{*}}}
{panel}
* check the configuration of the e1000g0 interface, especially the IP address, the netmask and the broadcast:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig e1000g0{*}}}
{{e1000g0: flags=201004843 mtu 1500 index 2}}
{{inet 10.6.90.104 netmask fffffe00 broadcast 10.6.91.255}}
{{ether 0:3:ba:d8:e1:c}}
{panel}
* configure vsw0 with the same parameters
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig vsw0 10.6.90.104 netmask 255.255.254.0 broadcast 10.6.91.255{*}}}
{panel}
* bring e1000g0 down
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig e1000g0 down{*}}}
{panel}
* bring vsw0 up
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig vsw0 up{*}}}
{panel}
* unplumb e1000g0
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig e1000g0 unplumb{*}}}
{panel}
To make these changes permanent reboot you need to rename the identification files as required for the new interface. These may depend on the naming services used. For example when using dhcp the files /etc/hostname.e1000g0 and /etc/dhcp.e1000g0 need to be changed:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *mv /etc/hostname.e1000g0 /etc/hostname.vsw0{*}}}
{{primary# *mv /etc/dhcp.e1000g0 /etc/dhcp.vsw0{*}}}
{panel}
h2. Creating private networks between domains
You can create a private network between domains by creating a virtual switch which is not associated with any physical network interface. For example, to create a private network between two guest domains ldg1 and ldg2:
* create a virtual switch with no physical network interface
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vsw vsw-private primary{*}}}
{panel}
* add a virtual network interface to ldg1 and ldg2 connected to that virtual switch
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vnet vnet-private vsw-private ldg1{*}}}
{{primary# *ldm add-vnet vnet-private vsw-private ldg2{*}}}
{panel}
Now the ldg1 and ldg2 can use their new network interfaces to communicate on a private network. These interfaces can not be used to communicate with the primary domain unless the vsw interface of vsw-private virtual switch is plumbed in the primary domain.
{note:title=Delayed Reconfiguration}
Note: with LDoms 1.1, the new virtual switch and vnet interfaces are immediately added to the domains after the add-vsw/add-vnet command. Before LDoms 1.1, the add-vsw/add-vnet command will trigger a delayed reconfiguration and the targeted domain has to be rebooted.
{note}
h1. Virtual Disks
h6. (Contributed by Alex Chartre)
h2. Vdisk Basics
A virtual disk is made of two components: the virtual disk itself as it appears in a domain guest, and the virtual disk backend which is where data will effectively be stored and where virtual I/Os will eventually end up. The virtual disk backend is exported from a service domain by the virtual disk server (vds) driver. The vds driver communicates with the virtual disk client (vdc) driver in the guest domain through the hypervisor using a logical domain channel (ldc). Finally a virtual disk appears as a /dev/[r]dsk/cXdYsZ device in the guest domain.
{panel:borderStyle=solid| borderColor=black|bgColor=white}
!vdisk_topology_v0.1.png!
h6. Virtual disk topology diagram
{panel}
The virtual disk backend can be a physical disk, a physical disk slice, a file or a volume from a volume management framework (like ZFS, SVM, VxVM...).
* A backend is exported from a domain using the command {{"ldm add vds-dev"}}:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev <backend> <volume_name>@<service_name>*}}
{panel}
* And it is assigned to another domain with the command {{"ldm add-vdisk"}}:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk <disk_name> <volume_name>@<service_name> <domain>*}}
{panel}
{note:Title=Domain Binding}
Note that a backend is effectively exported when the domain is bound.
{note}
h2. Virtual Disk Export Options
There are two ways a backend can be exported as a virtual disk, either as a *full disk* or as a *single slice disk*. The way the virtual disk depends on the type of the backend and on the options used to export it.
*Full Disk*
When a backend is exported to a domain as a full disk, it will appear in that domain as a regular disk with 8 slices (s0 to s7). Such a disk is visible with the format(1m) command and its partition table can be changed using either the fmthard(1m) or format(1m) command. A full disk will also be visible from the Solaris installer and can be selected as a disk device on which Solaris can be installed.
*Single Slice Disk*
Starting with Solaris 10 10/08 (U6), when a backend is exported to a domain as a single slice disk, it appears in that domain as a regular disk with 8 slices (s0 to s7). However only the first slice (s0) is usable. Such a disk is visible with the format(1M) command but the disk's partition table can not be changed. A single slice disk is also visible from the OS installation software and can be selected as a disk onto which the OS can be installed. In that case, if the OS installation is done using the UFS filesystem then only the root partition {{(/)}} must be defined and this partition must use all the disk space.
Before Solaris 10 10/08 (U6), when a backend is exported to a domain as a single slice disk, it will appear in that domain as a disk with a single partition (s0). Such a disk is not visible with the format(1m) command and its partition table can not be changed. A single slice disk will not be visible from the Solaris installer and can not be select as a disk device on which Solaris can be installed.
h2. Virtual Disk Backend
The virtual disk backend is the location where data of a virtual disk are effectively be stored. This backend can be a physical disk, a physical disk slice, a file or a volume (ZFS, SVM, VxVM...). The way a backend is exported (either as a full disk or as single slice disk) depends on the type of backend and on the Solaris release used in the service domain.
*Before Solaris 10 05/08 (Update 5) or patch 127127-11*
|| Backend || Export || Solaris Installation ||
| Disk (disk slice 2) | full disk | possible |
| Disk slice (not slice 2) | single slice disk | not possible |
| volume (ZFS, SVM, VxVM) | single slice disk | not possible |
*Since Solaris 10 05/08 (Update 5) or patch 127127-11*
Since Solaris 10 05/08 and LDoms 1.0.3, the "slice" option has been introduced as an optional parameter to the {{"ldm add-vdsdev"}} and {{"ldm set-vdsdev"}} commands:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev* *[options=slice]* *backend volume@service_name{*}}}
{panel}
A backend is normally exported either as a full disk or as a single slice disk depending on its type. If the slice option is specified, then the backend is forcibly exported as a single slice disk.
|| Backend || Export || Solaris Installation ||
| Disk (disk slice 2) | full disk (1) | single slice disk (2) |
| Disk slice (not slice 2) | single slice disk (3) | single slice disk |
| file | full disk | single slice disk |
| volume (ZFS, SVM, VxVM) | full disk | single slice disk |
(1) export the entire disk
(2) export only slice 2
(3) a slice is alwats exported as single slice disk
Before Solaris 10 10/08 (Update 6), the installation of Solaris is possible only on a full disk, not on a single-slice disk.
Starting with Solaris 10 10/08 (Update 6), the installation of Solaris is also possible on a single-slice disk. In that case, if the Solaris installation is done using the UFS filesystem then only the root partition {{(/)}} must be defined and this partition must use all the disk space.
h2. Physical Disks/LUNs
A physical disk is exported as a full disk. In that case, virtual disk drivers (vds and vdc) forward I/Os from the virtual disk and act as a pass-through to the physical disk. A physical disk can be exported by exporting the slice 2 (s2) of the disk.
*Example: exporting a physical disk as a virtual disk*
* To export the physical disk c1t48d0 as a virtual disk, we have to export the slice 2 of that disk (c1t48d0s2):
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /dev/dsk/c1t48d0s2 c1t48d0@primary-vds0{*}}}
{panel}
* Once the disk is exported, it can be assigned to a domain. Here it is assigned to the domain "test":
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk pdisk c1t48d0@primary-vds0 test{*}}}
{panel}
* Finally the disk is accessible from the guest domain "test" as a full disk (i.e. a regular disk with 8 slices); here the disk is accessible as c0d1:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d1s*\*}}
{{/dev/dsk/c0d1s0}}
{{/dev/dsk/c0d1s1}}
{{/dev/dsk/c0d1s2}}
{{/dev/dsk/c0d1s3}}
{{/dev/dsk/c0d1s4}}
{{/dev/dsk/c0d1s5}}
{{/dev/dsk/c0d1s6}}
{{/dev/dsk/c0d1s7}}
{panel}
h2. Physical Disk Slice
A physical disk slice is exported as a single slice disk. In that case, virtual disk drivers (vds and vdc) forward I/Os from the virtual disk and act as a pass-through to the physical disk slice.
*Example: exporting a physical disk slice as a virtual disk*
* To export the slice 0 of the physical disk c1t57d0 as a virtual disk, we have to export the device corresponding to that slice (c1t57d0s0):
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /dev/dsk/c1t57d0s0 c1t57d0s0@primary-vds0{*}}}
{panel}
* Once the disk is exported, it can be assigned to a domain. Here it is assigned to the domain "test" (Note the {{"pslice"}} option is used):
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk pslice c1t57d0s0@primary-vds0 test{*}}}
{panel}
* Finally the disk is accessible from the guest domain "test" as a single slice disk (i.e. a disk with only 1 slice: s0); here the disk is accessible as c0d13:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d13s*\*}}
{{/dev/dsk/c0d13s0\*}}
{panel}
{note:Title=Multiple Slice Support}
Note that starting with Solaris 10 10/08 (U6), a single slice disk appears with 8 slice devices. However only the first slice device (s0) is usable. This is so the Solaris installer can operate properly.
{note}
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d13s*\*}}
{{/dev/dsk/c0d13s0 <<-\- only use this slice}}
{{/dev/dsk/c0d13s1}}
{{/dev/dsk/c0d13s2}}
{{/dev/dsk/c0d13s3}}
{{/dev/dsk/c0d13s4}}
{{/dev/dsk/c0d13s5}}
{{/dev/dsk/c0d13s6}}
{{/dev/dsk/c0d13s7}}
{panel}
h2. Files and Volumes
A file or volume (for example from ZFS or SVM) is exported either as a full disk or as single slice disk depending on whether or not the slice option is set.
h3. File or Volume Exported as a Full Disk
If you do not set the slice option, a file or volume is exported as a full disk. In that case, virtual disk drivers (vds and vdc) forward I/O from the virtual disk and manage the partitioning of the virtual disk. The file or volume eventually becomes a disk image containing data from all slices of the virtual disk and the metadata used to manage the partitioning and disk structure.
When a blank file or volume is exported as full disk, it appears in the guest domain as an unformatted disk; that is, a disk with no partition. Then you need to run the format(1M) command in the guest domain to define usable partitions and to write a valid disk label. Any I/O to the virtual disk fails while the disk is unformatted.
Before the Solaris 10 5/08 OS release, when a blank file was exported as a virtual disk, the system wrote a default disk label and created default partitioning. This is no longer the case with the Solaris 10 5/08 OS release, and you must run format(1M) in the guest domain to create partitions.
*Example - exporting a file as a virtual disk*
To export the file /ldoms/domain/test/fdisk0 as a virtual disk, we first have to create it. The size of the file will define the size of the virtual disk.
* First we create a 100mb blank file to get a 100mb virtual disk:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *mkfile 100m /ldoms/domain/test/fdisk0{*}}}
{panel}
* Then the file can be directly exported as a virtual disk:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /ldoms/domain/test/fdisk0 fdisk0@primary-vds0{*}}}
{panel}
* Once the file is exported, it can be assigned to a domain. Here it is assigned to the domain "test":
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk fdisk fdisk0@primary-vds0 test{*}}}
{panel}
* Finally the disk is accessible from the guest domain "test" as a full disk (i.e. a regular disk with 8 slices); here the disk is accessible as c0d5:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d5s*\*}}
{{/dev/dsk/c0d5s0}}
{{/dev/dsk/c0d5s1}}
{{/dev/dsk/c0d5s2}}
{{/dev/dsk/c0d5s3}}
{{/dev/dsk/c0d5s4}}
{{/dev/dsk/c0d5s5}}
{{/dev/dsk/c0d5s6}}
{{/dev/dsk/c0d5s7}}
{panel}
h3. File or Volume Exported as a Single Slice Disk
If the slice option is set, then the file or volume is exported as a single slice disk. In that case, the virtual disk has only one partition (s0), which is directly mapped to the file or volume backend. The file or volume only contains data written to the virtual disk with no extra data like partitioning information or disk structure. When a file or volume is exported as a single slice disk, the system simulates a fake disk partitioning which makes that file or volume appear as a disk slice. Because the disk partitioning is simulated, you do not create a partitioning for that disk.
*Example - exporting an existing ZFS Volume as a Single Slice Disk*
Let us say you have an existing ZFS volume (tank/data) where you have stored data and you want to make these data available to a guest. In that case, you should export the ZFS volume as a single-slice disk.
* From the service domain, export the device corresponding to the ZFS volume, and set the slice option so that the volume is exported as a single slice disk.
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev options=slice /dev/zvol/dsk/tank/data zdisk@primary-vds0{*}}}
{panel}
* From the service domain, assign the volume (zdisk0) to the guest domain, ldg1 for example.
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk zdisk0 zdisk0@primary-vds0 ldg1{*}}}
{panel}
* After the guest domain is started and running the Solaris OS, the volume is accessible as single-disk (c0d9s0 for example).
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *ls \-1 /dev/dsk/c0d9s0{*}}}
{{/dev/dsk/c0d9s0}}
{panel}
h1. ZFS
h2. Cloning Disk Images
h6. (Adapted from an email by Peter A. Wilson)
You can use ZFS to very efficiently, easily and quickly, take a copy of a previously prepared "golden" boot disk for one domain and redeploy multiple copies of that image as a pre-installed boot disk for other domains.
This procedure takes advantage of the ZFS snapshotting and cloning facilities to replicate one disk for other domains, snapshotting and cloning of disk images is virtually instantaneous and consumes almost no additional space on the physical storage medium to to the way ZFS simply creates pointers to the original copy of the data for each clone of the image and will only require additional storage to keep track of deltas to the original disk image for each clone as they modify their own 'copy'.
*Procedure to create a gold image an then snapshot/clone for multiple domains*
* In the Control domain create a ZFS pool by assigning physical disk devices (spindles, slices, LUNs etc.)
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zpool create mypool c0t2d0 c0t3d0{*}}}
{panel}
* Create a ZFS volume in that pool
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zfs create mypool/domain1_virtual{*}}}
{panel}
* Create a file in the new volume large enough to contain the boot disk for the domain you will be setting up... (in this case 15GB)
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *mkfile 15g /mypool/domain1_virtual/disk_image{*}}}
{panel}
* Export that file as a virtual disk for the guest you are intending to set up as your golden image:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /mypool/domain1_virtual/disk_image vol1@primary_vds{*}}}
{{primary# *ldm add-vdisk vdisk0 vol1@primary_vds guest1{*}}}
{panel}
The guest will see that as a conventional scsi disk.
* Install Solaris in the domain "guest1" This can be done by any normal Solaris install means (jumpstart or with S10u5 and LDoms 1.0.3 via CDROM). Set up the OS in the guest as you require it, you can choose to then sys-unconfig that guest if you wish. Then stop and unbind the domain before cloning it.
* In the control domain use ZFS to snapshot and then clone the image:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zfs snapshot mypool/domain1_virtual@mycopy{*}}}
{{primary# *zfs clone mypool/domain1_virtual@mycopy mypool/domain2{*}}}
{panel}
The ZFS volume /mypool/domain2 will now contain a copy of the disk from domain1. This can then be assigned to another domain. Repeat this for as many guests as you want.
*Using a ZFS Volume rather than a file*
Here is the sequence of commands to do the same thing using a ZFS volume instead of a file. The advantage is that the creation of the ZFS volume is much faster than the creation of the file with mkfile:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zpool create mypool c0t2d0 c0t3d0{*}}}
{{primary# *zfs create mypool/domain1{*}}}
{{primary# *zfs create \-V 15g domain1/disk_image{*}}}
{{primary# *ldm add-vdsdev /dev/zvol/dsk/mypool/domain1/disk_image vol1@primary_vds{*}}}
{{primary# *ldm add-vdisk vdisk0 vol1@primary_vds guest1{*}}}
{{primary# *zfs snapshot mypool/domain1/diskimage@mycopy{*}}}
{{primary# *zfs create mypool/domain2{*}}}
{{primary# *zfs clone mypool/domain1/diskimage@mycopy mypool/domain2/diskimage{*}}}
{panel}
Then the ZFS volume mypool/domain2/diskimage will now contain a copy of the disk from domain2.
{note:Title=Notes}
* You can repeat the cloning as many times as you require for additional guest domains.
* You can check the extremely efficient use of disk space by checking the actual disk space used by these clones using the {{'zpool list'}} command.
{note}
h1. Solaris Volume Manager
Solaris Volume Manager (SVM) can be used in a guest domain with virtual disks like on a regular system.
If you want to create some SVM metaset in a guest domain then you have to use virtual disks which backend are physical SCSI disks or LUNs.
h1. Symantec VxVM
Symantec Storage Foundation 5.0MP3 which includes VxVM will be supported in guest domains with virtual disks. For more information see: [Symantec/Veritas Support Docs|http://www.symantec.com/business/support/documentation.jsp?language=english&view=manuals&pid=15107&version=FOUNDSUITEPVER27937]
h1. iSCSI
h2. Using iSCSI targets as vdisks
h6. (Adapted from an email by Martin Mueller)
The following is a simplified methodology to achieve iSCSI connectivity in an LDoms environment. It does not use any security mechanisms such as CHAP etc.
*Configure iSCSI Server & Client*
* Preparation of iSCSI targets on iSCSI server
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{iscsi_svr# *zfs create \-b 1024 \-o shareiscsi=on \-V 2G iscsi_test/vol1{*}}}
{panel}
* Prepare iSCSI client (initiator). Add iSCSI server to the discovery list
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *iscsiadm add discovery-address 10.16.11.181{*}}}
{panel}
* Enable discovery
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *iscsiadm add discovery-address 10.16.11.181{*}}}
{panel}
iSCSI target should now appear in format etc:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *format{*}}}
{{...}}
{{6. c4t0d0 SUN-SOLARIS-1 cyl 32766 alt 2 hd 4 sec 160}}
{panel}
*Configure logical domains*
* Put iSCSI target under vds control
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /dev/dsk/c4t0d0s2 boot-disk@disk-svc{*}}}
{panel}
The volume may now be assigned to guest domains as a vdisk as per normal LDoms procedures.
{column}
{section}
{column:width=25%}
h5. [LDoms Community Cookbook]
h6. Contents
{pagetree:root=@parent|searchBox=true}
h6. In this Section ...
{toc:type=list|style=none|maxLevel=3}
{column}
{column:width=75%}
h1. Section Introduction
This section provides background information on the virtual devices and services in Logical Domains, and provides examples on how to configure and use them.
----
h1. Virtual Networks
h2. Service Domain to Guest Domain Traffic
If you have a service domain with a virtual switch (for example primary-vsw0 in the primary) and guest domains using that virtual switch then, by default, guest domains will not be able to communicate with the service domain over the network.
To enable network communications between guest domains and a service domain then you need to plumb the interface of the virtual switch. So what you will usually do is plumb the virtual switch interface instead of the physical network interface the virtual switch is associated with.
For example, if your virtual switch is associated with the e1000g0 interface in the primary domain:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm ls \-l primary{*}}}
{{...}}
{{VSW}}
{{NAME MAC NET-DEV DEVICE DEFAULT-VLAN-ID PVID VID MODE}}
{{primary-vsw0 00:14:4f:fb:54:f2 e1000g0 switch@0 1 1}}
{{...}}
{panel}
Then you are going to plumb the vsw0 interface instead of e1000g0. The vsw0 interface should be configured with the same parameters (IP address, netmask, broadcast...) as the e1000g0 interface while the e1000g0 is going to be unplumbed.
{note:title=Use the Console!}
Switching from the e1000g0 interface to the vsw0 interface should be done from the console (or from an alternate network connection) as the e1000g0 network connection will be brought down during the switch.
{note}
* plumb the vsw interface
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig vsw0 plumb{*}}}
{panel}
* check the configuration of the e1000g0 interface, especially the IP address, the netmask and the broadcast:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig e1000g0{*}}}
{{e1000g0: flags=201004843 mtu 1500 index 2}}
{{inet 10.6.90.104 netmask fffffe00 broadcast 10.6.91.255}}
{{ether 0:3:ba:d8:e1:c}}
{panel}
* configure vsw0 with the same parameters
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig vsw0 10.6.90.104 netmask 255.255.254.0 broadcast 10.6.91.255{*}}}
{panel}
* bring e1000g0 down
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig e1000g0 down{*}}}
{panel}
* bring vsw0 up
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig vsw0 up{*}}}
{panel}
* unplumb e1000g0
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ifconfig e1000g0 unplumb{*}}}
{panel}
To make these changes permanent reboot you need to rename the identification files as required for the new interface. These may depend on the naming services used. For example when using dhcp the files /etc/hostname.e1000g0 and /etc/dhcp.e1000g0 need to be changed:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *mv /etc/hostname.e1000g0 /etc/hostname.vsw0{*}}}
{{primary# *mv /etc/dhcp.e1000g0 /etc/dhcp.vsw0{*}}}
{panel}
h2. Creating private networks between domains
You can create a private network between domains by creating a virtual switch which is not associated with any physical network interface. For example, to create a private network between two guest domains ldg1 and ldg2:
* create a virtual switch with no physical network interface
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vsw vsw-private primary{*}}}
{panel}
* add a virtual network interface to ldg1 and ldg2 connected to that virtual switch
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vnet vnet-private vsw-private ldg1{*}}}
{{primary# *ldm add-vnet vnet-private vsw-private ldg2{*}}}
{panel}
Now the ldg1 and ldg2 can use their new network interfaces to communicate on a private network. These interfaces can not be used to communicate with the primary domain unless the vsw interface of vsw-private virtual switch is plumbed in the primary domain.
{note:title=Delayed Reconfiguration}
Note: with LDoms 1.1, the new virtual switch and vnet interfaces are immediately added to the domains after the add-vsw/add-vnet command. Before LDoms 1.1, the add-vsw/add-vnet command will trigger a delayed reconfiguration and the targeted domain has to be rebooted.
{note}
h1. Virtual Disks
h6. (Contributed by Alex Chartre)
h2. Vdisk Basics
A virtual disk is made of two components: the virtual disk itself as it appears in a domain guest, and the virtual disk backend which is where data will effectively be stored and where virtual I/Os will eventually end up. The virtual disk backend is exported from a service domain by the virtual disk server (vds) driver. The vds driver communicates with the virtual disk client (vdc) driver in the guest domain through the hypervisor using a logical domain channel (ldc). Finally a virtual disk appears as a /dev/[r]dsk/cXdYsZ device in the guest domain.
{panel:borderStyle=solid| borderColor=black|bgColor=white}
!vdisk_topology_v0.1.png!
h6. Virtual disk topology diagram
{panel}
The virtual disk backend can be a physical disk, a physical disk slice, a file or a volume from a volume management framework (like ZFS, SVM, VxVM...).
* A backend is exported from a domain using the command {{"ldm add vds-dev"}}:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev <backend> <volume_name>@<service_name>*}}
{panel}
* And it is assigned to another domain with the command {{"ldm add-vdisk"}}:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk <disk_name> <volume_name>@<service_name> <domain>*}}
{panel}
{note:Title=Domain Binding}
Note that a backend is effectively exported when the domain is bound.
{note}
h2. Virtual Disk Export Options
There are two ways a backend can be exported as a virtual disk, either as a *full disk* or as a *single slice disk*. The way the virtual disk depends on the type of the backend and on the options used to export it.
*Full Disk*
When a backend is exported to a domain as a full disk, it will appear in that domain as a regular disk with 8 slices (s0 to s7). Such a disk is visible with the format(1m) command and its partition table can be changed using either the fmthard(1m) or format(1m) command. A full disk will also be visible from the Solaris installer and can be selected as a disk device on which Solaris can be installed.
*Single Slice Disk*
Starting with Solaris 10 10/08 (U6), when a backend is exported to a domain as a single slice disk, it appears in that domain as a regular disk with 8 slices (s0 to s7). However only the first slice (s0) is usable. Such a disk is visible with the format(1M) command but the disk's partition table can not be changed. A single slice disk is also visible from the OS installation software and can be selected as a disk onto which the OS can be installed. In that case, if the OS installation is done using the UFS filesystem then only the root partition {{(/)}} must be defined and this partition must use all the disk space.
Before Solaris 10 10/08 (U6), when a backend is exported to a domain as a single slice disk, it will appear in that domain as a disk with a single partition (s0). Such a disk is not visible with the format(1m) command and its partition table can not be changed. A single slice disk will not be visible from the Solaris installer and can not be select as a disk device on which Solaris can be installed.
h2. Virtual Disk Backend
The virtual disk backend is the location where data of a virtual disk are effectively be stored. This backend can be a physical disk, a physical disk slice, a file or a volume (ZFS, SVM, VxVM...). The way a backend is exported (either as a full disk or as single slice disk) depends on the type of backend and on the Solaris release used in the service domain.
*Before Solaris 10 05/08 (Update 5) or patch 127127-11*
|| Backend || Export || Solaris Installation ||
| Disk (disk slice 2) | full disk | possible |
| Disk slice (not slice 2) | single slice disk | not possible |
| volume (ZFS, SVM, VxVM) | single slice disk | not possible |
*Since Solaris 10 05/08 (Update 5) or patch 127127-11*
Since Solaris 10 05/08 and LDoms 1.0.3, the "slice" option has been introduced as an optional parameter to the {{"ldm add-vdsdev"}} and {{"ldm set-vdsdev"}} commands:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev* *[options=slice]* *backend volume@service_name{*}}}
{panel}
A backend is normally exported either as a full disk or as a single slice disk depending on its type. If the slice option is specified, then the backend is forcibly exported as a single slice disk.
|| Backend || Export || Solaris Installation ||
| Disk (disk slice 2) | full disk (1) | single slice disk (2) |
| Disk slice (not slice 2) | single slice disk (3) | single slice disk |
| file | full disk | single slice disk |
| volume (ZFS, SVM, VxVM) | full disk | single slice disk |
(1) export the entire disk
(2) export only slice 2
(3) a slice is alwats exported as single slice disk
Before Solaris 10 10/08 (Update 6), the installation of Solaris is possible only on a full disk, not on a single-slice disk.
Starting with Solaris 10 10/08 (Update 6), the installation of Solaris is also possible on a single-slice disk. In that case, if the Solaris installation is done using the UFS filesystem then only the root partition {{(/)}} must be defined and this partition must use all the disk space.
h2. Physical Disks/LUNs
A physical disk is exported as a full disk. In that case, virtual disk drivers (vds and vdc) forward I/Os from the virtual disk and act as a pass-through to the physical disk. A physical disk can be exported by exporting the slice 2 (s2) of the disk.
*Example: exporting a physical disk as a virtual disk*
* To export the physical disk c1t48d0 as a virtual disk, we have to export the slice 2 of that disk (c1t48d0s2):
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /dev/dsk/c1t48d0s2 c1t48d0@primary-vds0{*}}}
{panel}
* Once the disk is exported, it can be assigned to a domain. Here it is assigned to the domain "test":
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk pdisk c1t48d0@primary-vds0 test{*}}}
{panel}
* Finally the disk is accessible from the guest domain "test" as a full disk (i.e. a regular disk with 8 slices); here the disk is accessible as c0d1:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d1s*\*}}
{{/dev/dsk/c0d1s0}}
{{/dev/dsk/c0d1s1}}
{{/dev/dsk/c0d1s2}}
{{/dev/dsk/c0d1s3}}
{{/dev/dsk/c0d1s4}}
{{/dev/dsk/c0d1s5}}
{{/dev/dsk/c0d1s6}}
{{/dev/dsk/c0d1s7}}
{panel}
h2. Physical Disk Slice
A physical disk slice is exported as a single slice disk. In that case, virtual disk drivers (vds and vdc) forward I/Os from the virtual disk and act as a pass-through to the physical disk slice.
*Example: exporting a physical disk slice as a virtual disk*
* To export the slice 0 of the physical disk c1t57d0 as a virtual disk, we have to export the device corresponding to that slice (c1t57d0s0):
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /dev/dsk/c1t57d0s0 c1t57d0s0@primary-vds0{*}}}
{panel}
* Once the disk is exported, it can be assigned to a domain. Here it is assigned to the domain "test" (Note the {{"pslice"}} option is used):
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk pslice c1t57d0s0@primary-vds0 test{*}}}
{panel}
* Finally the disk is accessible from the guest domain "test" as a single slice disk (i.e. a disk with only 1 slice: s0); here the disk is accessible as c0d13:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d13s*\*}}
{{/dev/dsk/c0d13s0\*}}
{panel}
{note:Title=Multiple Slice Support}
Note that starting with Solaris 10 10/08 (U6), a single slice disk appears with 8 slice devices. However only the first slice device (s0) is usable. This is so the Solaris installer can operate properly.
{note}
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d13s*\*}}
{{/dev/dsk/c0d13s0 <<-\- only use this slice}}
{{/dev/dsk/c0d13s1}}
{{/dev/dsk/c0d13s2}}
{{/dev/dsk/c0d13s3}}
{{/dev/dsk/c0d13s4}}
{{/dev/dsk/c0d13s5}}
{{/dev/dsk/c0d13s6}}
{{/dev/dsk/c0d13s7}}
{panel}
h2. Files and Volumes
A file or volume (for example from ZFS or SVM) is exported either as a full disk or as single slice disk depending on whether or not the slice option is set.
h3. File or Volume Exported as a Full Disk
If you do not set the slice option, a file or volume is exported as a full disk. In that case, virtual disk drivers (vds and vdc) forward I/O from the virtual disk and manage the partitioning of the virtual disk. The file or volume eventually becomes a disk image containing data from all slices of the virtual disk and the metadata used to manage the partitioning and disk structure.
When a blank file or volume is exported as full disk, it appears in the guest domain as an unformatted disk; that is, a disk with no partition. Then you need to run the format(1M) command in the guest domain to define usable partitions and to write a valid disk label. Any I/O to the virtual disk fails while the disk is unformatted.
Before the Solaris 10 5/08 OS release, when a blank file was exported as a virtual disk, the system wrote a default disk label and created default partitioning. This is no longer the case with the Solaris 10 5/08 OS release, and you must run format(1M) in the guest domain to create partitions.
*Example - exporting a file as a virtual disk*
To export the file /ldoms/domain/test/fdisk0 as a virtual disk, we first have to create it. The size of the file will define the size of the virtual disk.
* First we create a 100mb blank file to get a 100mb virtual disk:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *mkfile 100m /ldoms/domain/test/fdisk0{*}}}
{panel}
* Then the file can be directly exported as a virtual disk:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /ldoms/domain/test/fdisk0 fdisk0@primary-vds0{*}}}
{panel}
* Once the file is exported, it can be assigned to a domain. Here it is assigned to the domain "test":
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk fdisk fdisk0@primary-vds0 test{*}}}
{panel}
* Finally the disk is accessible from the guest domain "test" as a full disk (i.e. a regular disk with 8 slices); here the disk is accessible as c0d5:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{test# *ls \-1 /dev/dsk/c0d5s*\*}}
{{/dev/dsk/c0d5s0}}
{{/dev/dsk/c0d5s1}}
{{/dev/dsk/c0d5s2}}
{{/dev/dsk/c0d5s3}}
{{/dev/dsk/c0d5s4}}
{{/dev/dsk/c0d5s5}}
{{/dev/dsk/c0d5s6}}
{{/dev/dsk/c0d5s7}}
{panel}
h3. File or Volume Exported as a Single Slice Disk
If the slice option is set, then the file or volume is exported as a single slice disk. In that case, the virtual disk has only one partition (s0), which is directly mapped to the file or volume backend. The file or volume only contains data written to the virtual disk with no extra data like partitioning information or disk structure. When a file or volume is exported as a single slice disk, the system simulates a fake disk partitioning which makes that file or volume appear as a disk slice. Because the disk partitioning is simulated, you do not create a partitioning for that disk.
*Example - exporting an existing ZFS Volume as a Single Slice Disk*
Let us say you have an existing ZFS volume (tank/data) where you have stored data and you want to make these data available to a guest. In that case, you should export the ZFS volume as a single-slice disk.
* From the service domain, export the device corresponding to the ZFS volume, and set the slice option so that the volume is exported as a single slice disk.
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev options=slice /dev/zvol/dsk/tank/data zdisk@primary-vds0{*}}}
{panel}
* From the service domain, assign the volume (zdisk0) to the guest domain, ldg1 for example.
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdisk zdisk0 zdisk0@primary-vds0 ldg1{*}}}
{panel}
* After the guest domain is started and running the Solaris OS, the volume is accessible as single-disk (c0d9s0 for example).
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{ldg1# *ls \-1 /dev/dsk/c0d9s0{*}}}
{{/dev/dsk/c0d9s0}}
{panel}
h1. ZFS
h2. Cloning Disk Images
h6. (Adapted from an email by Peter A. Wilson)
You can use ZFS to very efficiently, easily and quickly, take a copy of a previously prepared "golden" boot disk for one domain and redeploy multiple copies of that image as a pre-installed boot disk for other domains.
This procedure takes advantage of the ZFS snapshotting and cloning facilities to replicate one disk for other domains, snapshotting and cloning of disk images is virtually instantaneous and consumes almost no additional space on the physical storage medium to to the way ZFS simply creates pointers to the original copy of the data for each clone of the image and will only require additional storage to keep track of deltas to the original disk image for each clone as they modify their own 'copy'.
*Procedure to create a gold image an then snapshot/clone for multiple domains*
* In the Control domain create a ZFS pool by assigning physical disk devices (spindles, slices, LUNs etc.)
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zpool create mypool c0t2d0 c0t3d0{*}}}
{panel}
* Create a ZFS volume in that pool
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zfs create mypool/domain1_virtual{*}}}
{panel}
* Create a file in the new volume large enough to contain the boot disk for the domain you will be setting up... (in this case 15GB)
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *mkfile 15g /mypool/domain1_virtual/disk_image{*}}}
{panel}
* Export that file as a virtual disk for the guest you are intending to set up as your golden image:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /mypool/domain1_virtual/disk_image vol1@primary_vds{*}}}
{{primary# *ldm add-vdisk vdisk0 vol1@primary_vds guest1{*}}}
{panel}
The guest will see that as a conventional scsi disk.
* Install Solaris in the domain "guest1" This can be done by any normal Solaris install means (jumpstart or with S10u5 and LDoms 1.0.3 via CDROM). Set up the OS in the guest as you require it, you can choose to then sys-unconfig that guest if you wish. Then stop and unbind the domain before cloning it.
* In the control domain use ZFS to snapshot and then clone the image:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zfs snapshot mypool/domain1_virtual@mycopy{*}}}
{{primary# *zfs clone mypool/domain1_virtual@mycopy mypool/domain2{*}}}
{panel}
The ZFS volume /mypool/domain2 will now contain a copy of the disk from domain1. This can then be assigned to another domain. Repeat this for as many guests as you want.
*Using a ZFS Volume rather than a file*
Here is the sequence of commands to do the same thing using a ZFS volume instead of a file. The advantage is that the creation of the ZFS volume is much faster than the creation of the file with mkfile:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *zpool create mypool c0t2d0 c0t3d0{*}}}
{{primary# *zfs create mypool/domain1{*}}}
{{primary# *zfs create \-V 15g domain1/disk_image{*}}}
{{primary# *ldm add-vdsdev /dev/zvol/dsk/mypool/domain1/disk_image vol1@primary_vds{*}}}
{{primary# *ldm add-vdisk vdisk0 vol1@primary_vds guest1{*}}}
{{primary# *zfs snapshot mypool/domain1/diskimage@mycopy{*}}}
{{primary# *zfs create mypool/domain2{*}}}
{{primary# *zfs clone mypool/domain1/diskimage@mycopy mypool/domain2/diskimage{*}}}
{panel}
Then the ZFS volume mypool/domain2/diskimage will now contain a copy of the disk from domain2.
{note:Title=Notes}
* You can repeat the cloning as many times as you require for additional guest domains.
* You can check the extremely efficient use of disk space by checking the actual disk space used by these clones using the {{'zpool list'}} command.
{note}
h1. Solaris Volume Manager
Solaris Volume Manager (SVM) can be used in a guest domain with virtual disks like on a regular system.
If you want to create some SVM metaset in a guest domain then you have to use virtual disks which backend are physical SCSI disks or LUNs.
h1. Symantec VxVM
Symantec Storage Foundation 5.0MP3 which includes VxVM will be supported in guest domains with virtual disks. For more information see: [Symantec/Veritas Support Docs|http://www.symantec.com/business/support/documentation.jsp?language=english&view=manuals&pid=15107&version=FOUNDSUITEPVER27937]
h1. iSCSI
h2. Using iSCSI targets as vdisks
h6. (Adapted from an email by Martin Mueller)
The following is a simplified methodology to achieve iSCSI connectivity in an LDoms environment. It does not use any security mechanisms such as CHAP etc.
*Configure iSCSI Server & Client*
* Preparation of iSCSI targets on iSCSI server
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{iscsi_svr# *zfs create \-b 1024 \-o shareiscsi=on \-V 2G iscsi_test/vol1{*}}}
{panel}
* Prepare iSCSI client (initiator). Add iSCSI server to the discovery list
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *iscsiadm add discovery-address 10.16.11.181{*}}}
{panel}
* Enable discovery
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *iscsiadm add discovery-address 10.16.11.181{*}}}
{panel}
iSCSI target should now appear in format etc:
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *format{*}}}
{{...}}
{{6. c4t0d0 SUN-SOLARIS-1 cyl 32766 alt 2 hd 4 sec 160}}
{panel}
*Configure logical domains*
* Put iSCSI target under vds control
{panel:borderStyle=solid| borderColor=black|bgColor=#e6e6e6}
{{primary# *ldm add-vdsdev /dev/dsk/c4t0d0s2 boot-disk@disk-svc{*}}}
{panel}
The volume may now be assigned to guest domains as a vdisk as per normal LDoms procedures.
{column}
{section}