LDoms Community Cookbook

Contents

In this Section ...

Section Introduction

(Contributed by Alexandre Chartre, Tony Shoumack, Pallab Bhattacharya, Sudhir Bhole)

This section provides details on the hardware platforms that can run Logical Domains and their capabilities. It also provide guidelines on how domains can be sized and how to select an appropriate system.

Sizing Guidelines

guideline, depends on the type of domains.

How many domains should be created?

The number of domains created will mostly depend on the amount of resources (cpus, memory, physical I/O) assigned to each domain, and the total amount of resources available on the system.

In any case, you will have a minimum of one domain, the primary domain. The primary domain is the control domain and it is also an I/O domain. In addition, depending on your platform and the physical resources available on the system, you can create additional I/O domains (up to a total of 4 I/O domains on the Sun SPARC Entreprise T5440 server). I/O domains can be used as standalone domains to run application requiring direct access to physical I/O devices, or as service domains to provide virtual device services (such as virtual disk or virtual network) to guest domains. Finally guest domains can be added up to the maximum number of domains supported on the system or to the amount of physical cpu threads or memory available.

How much memory should be assign to each domain?

The amount of memory required for each domain mostly depends on applications and workload running on that domain. The Solaris Operating System requires a minimum of 1GB of memory so that will the minimum amount of memory to assign to one domain.

• A service domain will provide virtual device services to some other domains, and it will have to process any virtual I/O (such as virtual disk or virtual network requests) for domains it is servicing. As such it is more likely to have a more important workload, especially if applications running in other domains are very I/O intense. The guideline in that case is to assign a minimum of 2GB of memory to a service domain.

• An I/O domain does not have any special memory requirement, so the amount of memory required depends on the application and workload used, with a minimum of 1GB of memory for running the Solaris OS. However if an I/O domain is also used a service domain then the memory requirement for a service domain has to be applied (i.e. minimum 2GB of memory).

• The control domain is an I/O domain, and it does not have any additional memory requirement. So the amount of memory required also depends on the application and workload used, with a minimum of 1GB of memory for running the Solaris OS. But, similarly, if the control domain is used a service domain then the memory requirement for a service domain has to be applied (i.e. minimum 2GB of memory).

The amount of memory required depends on the running system and appplications The amount of memory required mostly depends on the behavior of the system and applications running on the domain. An illustration is a domain using ZFS with Solaris 10 11/06 (Update 3) or before or without patch 120011-14. In that case, ZFS can consume a lot of memory for caching information. This is particularly true if the domain is a service domain exporting ZFS file or volume as virtual disks. In that case, the guideline is to use a minimum of 4GB of memory to overcome the misbehavior of ZFS.

The amount of memory assigned to a domain can be changed after the domain has been created. However, the LDoms software does not currently support dynamic reconfiguration of memory, so changing the amount of memory assigned to an active domain requires that domain to be restarted for the change to be effective.

How many cpus should be assign to each domain?

The number of cpus required for each domain mostly depends on applications and workload running on that domain. However a general good practice is to assign entire cpu cores to domains. So the minimum guideline is to assign a minimum of one cpu core to each domain (note that depending on the cpu chip and system type, the number of cpu threads per cpu core can be different).

• A service domain will provide virtual device services to some other domains, and it will have to process any virtual I/O (such as virtual disk or virtual network requests) for domains it is servicing. As such it is more likely to have a more important workload, especially if applications running in other domains are very I/O intensive. The guideline in that case is to assign a minimum of two cpu cores to a service domain.

• An I/O domain does not have any special memory requirement, so the amount of memory required depends on the application and workload used, with a minimum of one cpu core. However if an I/O domain is also used a service domain then the memory requirement for a service domain has to be applied (i.e. minimum two cpu cores). Also if an I/O domain has a lot of I/O devices then the number of cpus might have to be increase to better handle the I/O requests (for example, an I/O domain with a lot of 10GB Ethernet interfaces can have a lot of network interrupts to process and require more cpus).

• The control domain is an I/O domain, and it does have not any additional cpu requirement. So the number of cpu required also depends on the application and workload used, with a minimum of one cpu core. But, similarly, if the control domain is used a service domain then the memory requirement for a service domain has to be applied (i.e. minimum two cpu core).

The amount of virtual cpus assigned to a domain can be changed after the domain has been created. Virtual cpus can be dynamically added or removed from a domain without having to stop, restart or interrupt the domain. So you might want to provision additional cpus so that you can later add more cpus to a domain if, for example, it has a more important workload.

How many I/O domain should be used?

Any system will have a minimum of one I/O domain which is the primary domain (it is also the control domain). You might want to create additional I/O domains in order to:

• provide service domain redundancy: an I/O domain is often used as a service domain to provide virtual device services (virtual disk, virtual network) to some guest domains. When that service domain is not available then the dependent guest domains can not access their virtual devices and can be partially hang until the service domain becomes available again. This situation can be avoiding by having multiple (usually 2) I/O domains acting as services domains and providing virtual device services to the same guest domains, and by also having guest domains used multipathing solutions to access their virtual devices through the different service domains. That way if a service domain becomes unavailable then guest domains can still access their virtual devices through another service domain.

• isolate the control domain: from a security standpoint, the control domain is a critical domain because it has a privileged connection with the hypervisor, and this is the only domain from where you can control all other domains and the entire platform. So you might want to create an additional I/O domain in order to isolate the control domain and restrict the access, workload and applications to the minimum required for the system management. Then the additional I/O domain can be used as a service domain (instead of using the control domain) for guest domains. That way the control domain is different from the service domain, and any problem on the service domain will not compromise the entire platform.

Also an I/O domain also has some capabilities which are not available in a guest domain using virtual devices, such as:

• optimal I/O performances: an I/O domain has direct access to the physical devices it owns (disks, network interfaces...) and uses the native (i.e. non-virtualized) drivers of the operating system. So an I/O domain will have the same I/O performances as a bare metal system (i.e. a system not configured with LDoms); this can be useful when running applications doing a lot of I/O or requiring optimal I/O performances.
  A guest domain using virtual devices will have a slight overhead in processing I/Os because I/Os also have to be processed by the service domain. The amount of overhead added depends on several factors, such as the configuration of the service and guest domains (number of cpus, memory size...), the workload in both domains and the physical device associated with the virtual device.

• access to tape devices, usb devices, frame buffer: currently the LDoms software does not virtualize tape devices, usb devices or frame buffers, so such a device can not be accessed from a guest domain. However tape devices, usb devices or frame buffers can be accessed from an I/O domain as long as it is connected to that I/O domain.

So if you want to use such capabilities, you can either use them from the primary domain (which is an I/O domain), or create another I/O domain if you want to have an isolation from the other domains.

Domain Migration Domain migration only works with guest domains and virtual devices, so if you want to be able to migrate a particular domain then that domain should not be an I/O domain.

Do I need external any external storage array/SAN?

If you want to have multiple I/O domains then you will need some external storage array or SAN connected to your system. With multiple I/O domains, only one I/O domain will have access to the internal disks present on the system. So, in order to provide disk storage to a second I/O domain, an external storage array or SAN has to be connected to that I/O domain.

If you plan to have only one I/O domain (the primary domain) then the LDoms configuration has no requirement for an external storage array or NAS.

Do I need network hybrid I/O?

If you want to have bare metal network performances in a guest domain (i.e. performances similar to the performances of a physical network interface) then you will need to use network hybrid I/O.

Sizing guidelines summary

The following table indicate some minimum guideline to size a domain depending on its role. If a domain assumes several roles (for example a control domain which is also an I/O domain and a service domain) then the most restrictive guideline should be selected.

(*) if the domain is also a service domain then the guideline for the service domain has to be selected

Minimum Guidelines These guidelines are just minimal values, they have to be adjusted depending on applications and workload running on the domain.

Choosing the Right System

Select a type of system

The most common type of system is the traditional standalone server (Sun SPARC Enterprise Servers). Alternatively if you have an expandable system, like the Sun Blade 6000 or 6048 Modular System, you might want to consider Sun Blade server modules. If you need a carrier-grade server then you should look at Sun Netra servers and blades.

• Sun SPARC Enterprise Servers: traditional standalone servers also known as CoolThreads servers.
  Sun CoolThreads Product Page

• Sun Blade Server Module: blade module for the modular Sun Blade 6000 family.

• Sun Netra Servers: carrier-grade rackmount servers for Network Equipment Providers (NEPs) and Service Providers (SPs) in the Communication and Media industry.

• Sun Netra Blade: carrier-grade ACTA blade for Network Equipment Providers (NEPs) and Service Providers (SPs) in the Communication and Media industry. Provide ultra-dense horizontal scalability, allowing to mix a wide range of processors and operating systems within a blade chassis, delivering flexibility and built-in 10G switch fabric support.
  Sun Netra Product Page

Check the number of domains

Each system has a limit on the maximum domain you can create, this varies from 32 up to 128. However a performance recommendation is to add an entire cpu core to each domain, so with this recommendation the maximum number of domains is limited by the number of cpu cores of the system, which varies from 8 to 32.

Check the number of cpu cores and threads

Each domain is assigned a certain number of virtual cpus, one virtual cpu corresponding to one cpu thread. Two domains can not share the same virtual cpu. So you have to compare the total number of virtual cpus you want to assign to all domains with the number of cpu threads available on the system.

LDoms supports cpu dynamic reconfiguration which can be used to dynamically add or remove virtual cpus to an active. This is useful, for example, to balance virtual cpus between the different domains depending of the workload of each domain. In that case, you have to take into account the maximum number of virtual cpus you want to be able to assign to each domain.

Check the memory size

Depending on the system, the maximum amount of memory available varies from 32GB to 512GB. Each domain can be assigned a chunk of the physical memory available, and this chunk of memory can not be shared. Also the hypervisor will reserve a small part of the memory (less than 512MB) that can not be allocated to any domain. So you have to compare the total amount of memory you want to assign to all domains plus the memory allocated by the hypervisor (you can it round up to 512MB) with the size of the memory available on the system.

Check I/O domains capabilities

If you want to create multiple I/O domains then you have to carefully select the more appropriate system. The number of I/O domains you can create depends on the number of PCI buses available on a system. Basically you can create one I/O domain per PCI bus, and the I/O domain will have access to the devices connected to the PCI bus assign to it. Some systems have only one PCI bus, so they support only one I/O domain (in that case the primary domain). Several systems have two PCI buses, so they support up to two I/O domains. If you want more than two I/O domains then you need to select the Sun SPARC Enterprise T5440 Server which has 4 PCI buses and support up to 4 I/O domains.

In addition to checking the number of PCI buses on a system, you should also pay attention to the layout of the PCI buses and check which on-board devices (internal disks, on-board network interfaces), usb ports and PCI slots are connected to the different PCI buses. This is important in order to be able to connect devices you want to a particular PCI bus so that they are accessible from a given I/O domain. Also when creating multiple I/O domains, some I/O domains will not have access to onboard the network interfaces or to the internal disks, so you will have to add some PCI cards (to the appropriate PCI bus) to provide disk and network access to those I/O domains.

Several sections below detail the PCI buses layout and capabilities for different systems.

Check network hybrid I/O capabilities

If you want to have network hybrid I/O capability when you need to select an UltraSPARC-T2 based system. Currently only UltraSPARC-T2 based systems support network hybrid I/O.

Network Hybrid I/O on UltraSPARC-T2 Plus Systems Although the UltraSPARC-T2 Plus processor has capabilities very similar to the UltraSPARC-T2 processor, UltraSPARC-T2 Plus based systems do not currently support network hybrid I/O.

Check floating point requirements

The UltraSPARC-T1 processor shares a single floating point unit among all cpu cores, so UltraSPARC-T1 based systems may not be suitable for workload doing a lot of floating point operations. High floating point content is not an issue for UltraSPARC-T2 and UltraSPARC-T2 Plus processors because these processors have a floating point unit per cpu core (each floating point unit is shared by all cpu threads in the cpu core).

You can use the CoolThreads Selection Tool (cooltst) to check evaluate the floating point content of a workload.

Systems Overview

In this section we will examine topologies of the systems under examination in this document, and their affect on various aspects of Logical Domains functionality. We will pay particular attention to the bus architectures and the implications for partitioning physical devices with LDoms. As systems may be available in various combinations and quantities of componentry, we will not be able to cover in detail all permutations. We will instead cover the most common configurations of each system.

Sun Logical Domains is designed to run on the sun4v-based systems, often referred to as "Chip Multi-Threading" or CMT systems. These systems have the UltraSPARC "T series" chips which contain multiple cores, with many threads-per-core. Each of the various generations of systems and processors differ in configuration and hence capabilities in terms of Logical Domains. The following is a listing of some of the currently available systems and their maximum LDoms capabilities.

The following tables summarize the main characteristics of servers supporting Logical Domains. Each table provides the following information.

• System: name of the system.
• CPU: type of cpu chip present on the system:
  • T1: UltraSPARC-T1, the first generation of Chip Multi-Threading (CMT) cpu. A single cpu chip has up to 8 cpu cores and 4 cpu threads per core, for a maximum of 32 cpu threads.
  • T2: UltraSPARC-T2, the second generation of CMT cpu. A single cpu chip has up to 8 cpu cores and 8 cpu threads per core, for a maximum of 64 cpu threads.
  • T2+: UltraSPARC-T2 Plus, an upgraded version of the UltraSPARC-T2. It provides the same number of cpu cores and threads as the UltraSPARC-T2 (up to 64 cpu threads) but several cpu chips can be installed on a single server providing up to 256 cpu threads.
• #Sockets: number of cpu sockets present on the system. This indicates the maximum number of cpu chips that can be installed. UltraSPARC-T1 and T2 based systems only support one socket, but UltraSPARC-T2 Plus based systems can have multiple sockets.
• #Cores: maximum number of cpu cores possible on the system. This is equal to #Socket * (maximum number of cores for the cpu chip).
• #Threads: maximum number of cpu threads possible on the system. This is equal to #Core * (maximum number of threads per core).
• Memory: maximum amount of memory possible on the system.
• I/O Bus: number of PCI buses on the system.
• Hybrid I/O: indicate if the system supports network hybrid I/O.
• Domains: maximum number of domains supported on the system. The number is given for the latest version of the LDoms software, it may be smaller if you are using another version.

Sun SPARC Enterprise Server

Number of I/O Domains on the Sun SPARC Enterprise T1000 Server Although the Sun SPARC Enterprise T1000 Server has two PCI buses, you can not create two fully operational I/O domains with both I/O domains having network and disk acccess.

Sun Blade Server Module

Sun Netra Server

Sun Netra Blade

Sun SPARC Enterprise Server Details

Sun SPARC Enterprise T1000 Server

Description

The Sun SPARC Enterprise T1000 Server (formerly named Sun Fire T1000 Server) is the entry-level UltraSPARC-T1 processor based server, which is optimized for heavily threaded workloads including, but not limited to web servers, service oriented architectures, application development and portals. The Sun SPARC Enterprise T1000 Server also offers extensive RAS features and unique power efficiency.

Sun Product Page

Bus Topology

• Number of PCI buses: 2
  • bus_a (pci@780)
    • PCI-E slot 0
  • bus_b (pci@7c0)
    • 1 internal SATA disk drives or 2 internal SAS disk drives
    • 4 onboard gigabit ethernet interfaces

Although the system has two PCI buses, it is not very appropriate for creating multiple I/O domains because bus_a has no other device but a single PCI slot.
This slot can be filled either with:

• a network card that would allow the creation of an I/O domain using bus_a which has a network access but no disk;
• or with an HBA card that would allow the creation of an I/O domain using bus_a which has a local disk but no network access.

As most systems require to have both a local disk and a network access, it is not possible to create an I/O domain using bus_a and therefore to have two usuable I/O domains on this system.

Sun SPARC Enterprise T2000 Server

Description

The Sun SPARC Enterprise T2000 Server (formerly named Sun Fire T2000 Server) is the first UltraSPARC-T1 based system which is optimized for multi-threaded workloads combined with maximum power efficiency. The system has two PCI buses which allow the creation of two I/O domains.

Sun Product Page

Bus Topology

• Number of PCI buses: 2
  • bus_a (pci@780)
    • 4 internal SAS disk drives (RoHS system)
    • 2 onboard gigabit interfaces (interace 0 and 1)
    • 1 PCI-E slot (slot 0)
  • bus_b (pci@7c0)
    • 1 internal CD/DVD drive
    • 4 internal SAS disk drives (non-RoHS system)
    • 2 onboard gigabit interfaces (interface 2 and 3)
    • 2 PCI-E slots (slot 1 and 2)
    • 2 PCI-X slots (slot 0 and 1)
    • 4 USB ports

Internal SAS Disk Drives Location The location of the 4 internal SAS disk drives depends on the type of systems: RoHS systems (part number 501-7501) have internal disks on bus_a Non-RoHS systems (part number 501-6843) have internal disks on bus_b

Sun SPARC Enterprise T5120 Server

Description

The Sun SPARC Enterprise T5120 and T5220 Server are the first systems based on the UltraSPARC-T2 Chip. The UltraSPARC-T2 chip doubles the threads from 32 to 64 as compared to the UltraSPARC-T1 chip. It also eliminates the Floating Point limitation in the UltraSPARC-T1 chip by having one floating point unit per core on the UltraSPARC-T2 chip. These servers only have one PCI bus, however they have XAUI ports which can host network adapters that can be used in hybrid I/O mode.

Sun Product Page

Bus Topology

• Number of PCI buses: 1
  • bus_a (pci@0)
    • 1 internal CD/DVD drive
    • 4 internal SAS disk drives
    • 4 onboard gigabit interfaces
    • 3 PCI-E slots
    • 2 XAUI ports (same slots as PCI-E slots 0 and 1)
    • 4 USB ports

Sun SPARC Enterprise T5220 Server

Description

The Sun SPARC Enterprise T5120 and T5220 Server are the first systems based on the UltraSPARC-T2 Chip. The UltraSPARC-T2 chip doubles the threads from 32 to 64 as compared to the UltraSPARC-T1 chip. It also eliminates the Floating Point limitation in the UltraSPARC-T1 chip by having one floating point unit per core on the UltraSPARC-T2 chip. These servers only have one PCI bus, however they have XAUI ports which can host network adapters that can be used in hybrid I/O mode.

Sun Product Page

Bus Topology

• Number of PCI buses: 1
  • bus_a (pci@0)
    • 1 internal CD/DVD drive
    • 8 internal SAS disk drives
    • 4 onboard gigabit interfaces
    • 6 PCI-E slots
    • 2 XAUI ports (same slots as PCI-E slots 0 and 1)
    • 4 USB ports

Sun SPARC Enterprise T5140 Server

Description

The Sun SPARC Enterprise T5140 Server begins the next wave of high-efficiency systems based on the second generation of Sun's Chip Multi-threaded Technology (CMT). It follows the previously announced Sun SPARC Enterprise T5120 and T5220 servers, with which it shares numerous features. The Sun SPARC Enterprise T5140 server utilizes multiple UltraSPARC-T2 Plus processors.

As the system has multiple chips, it uses cache coherency protocols via connections between the chips to manage the sharing of data.

Sun Product Page

Bus Topology

• Number of PCI buses: 2
  • bus_a (pci@200)
    • 1 internal CD/DVD drive
    • 4 internal SAS disk drives
    • 2 PCI-E slots (slots 1 and 2)
    • 4 USB ports
  • bus_b (pci@300)
    • 4 onboard gigabit interfaces
    • 1 PCI-E slot (slot 0)
    • 2 XAUI ports (same slots as PCI-E slots 0 and 1)

PCI-E and XAUI Slot 1 The PCI-E/XAUI slot 1 is either attached to bus_a or bus_b depending on which type of card is inserted in that slot: it is attached to bus_a if a PCI-E card is inserted, or it is attached to bus_b if an XAUI card is inserted.

Network Hybrid I/O Although the server has a Network Interface Unit (NIU) and XAUI slots, it does not currently support network hybrid I/O.

Sun SPARC Enterprise T5240 Server

Description

Sun Product Page

Bus Topology

• Number of PCI buses: 2
  • bus_a (pci@200)
    • 1 internal CD/DVD drive
    • 16 internal SAS disk drives
    • 3 PCI-E slots (slots 1 to 3)
    • 4 USB ports
  • bus_b (pci@300)
    • 4 onboard gigabit interfaces
    • 3 PCI-E slots (slots 0, 4 and 5)
    • 2 XAUI ports (same slots as PCI-E slots 0 and 1)

PCI-E and XAUI Slot 1 The PCI-E/XAUI slot 1 is either attached to bus_a or bus_b depending on which type of card is inserted in that slot: it is attached to bus_a if a PCI-E card is inserted, or it is attached to bus_b if an XAUI card is inserted.

Network Hybrid I/O Although the server has a Network Interface Unit (NIU) and XAUI slots, it does not currently support network hybrid I/O.

Sun SPARC Enterprise T5440 Server

Description

The Sun SPARC Enterprise T5440 Server is the largest member of the UltraSPARC-T2 Plus processor based SMP T5x40 product family. The server is optimized for heavily threaded workloads including, but not limited to web servers, service oriented architectures, application servers, portals and databases. The Sun SPARC Enterprise T5440 Server is housed in the 4RU brushed aluminum chassis also used in the x86 product line. The server has 4 SAS disks (73 or 146GB 10k RPM or 73GB 15k RPM drives), offers a total of 512GB memory in its 64 DIMM sockets and 8 PCI-Express slots. Four of the PCI-Express slots are x8, two slots are x8 with a x16 connector and two slots are x8 and shared with the XAUI 10Gbps Ethernet interfaces. The server also has four gigabit Ethernet? interfaces on the motherboard and offers redundant power and cooling via its four, hot-pluggable power supplies.

A Sun SPARC Enterprise T5440 Server comes with 4 UltraSPARC-T2 Plus installed on it. Each of the UltraSPARC-T2 Plus is directly connected to ¼th of the entire system memory with one gigabyte memory interleaving and owns a PCIe Root-Complex.

Sun Product Page

Bus Topology

• Number of PCI buses: 4
  • bus_a (pci@400)
    • 1 internal CD/DVD drive
    • 4 internal SAS disk drives
    • 2 PCI-E slots (slots 0 and 1)
    • 4 USB ports
  • bus_b (pci@600)
    • 2 PCI-E slots (slots 2 and 3)
  • bus_c (pci@500)
    • 4 onboard gigabit interfaces
    • 2 PCI-E slots (slots 4 and 5)
    • 2 XAUI ports (same slots as PCI-E slots 4 and 5)
  • bus_d (pci@700)
    • 2 PCI-E slots (slots 6 and 7)

Network Hybrid I/O Although the server has a Network Interface Unit (NIU) and XAUI slots, it does not currently support network hybrid I/O.