 | Be Careful This is a Storage System Pattern that is in progress. The contents are unstable as the system is being built. Please feel free to help build the pattern / implementation steps if you have some time. |
Overview
Small companies, workgroups within Enterprises, and individuals purchase servers and build them to be all-in-one boxes for delivering participation infrastructures to their target audiences. There are a variety of problems with a single box approach:
- Underutilized in its early deployment
- Overutilized if the deployment is successful
- Different aspects of the deployment will become overutilized at different rates (storage, CPU, bandwidth)
- This standalone box often becomes a maintenance headache
If the deployment is successful, over time, the server will become underpowered in one or more ways. The quickest path to upgrade is often to commission a completely new box with more CPU, Storage and Bandwidth and deploy the entire infrastructure to the new box (vertical scaling). Another approach may be to break the pieces apart (as shown below) on the participation infrastructure and grow them independently (a "deconsolidation" strategy). As the software was originally assembled to be hosted on a single box, this can be a relatively painful task.
The single box approach does yield sensible economics. But, with a little ingenuity, we can build this single box in such a way that it can easily be a multi-purpose box while the participation infrastructure is in its underutilized phase. Further, we can facilitate the partitioning of the software onto multiple boxes (horizontal scaling) or the movement to a larger box (vertical scaling) in a much more seamless fashion.
This Blue Print seeks to build a Storage System that optimizes on these criteria:
- Maximum utilization of applications on the Storage System when the overall participation infrastructure is underutilized for that purpose
- Ease of partitioning onto multiple Systems when the Storage System becomes overutilized in some aspect
- Ease of transition to larger systems if vertical scaling is required
Requirements
- All applications for a participation infrastructure run on a single server
- Applications are partitioned from one another so that if one misbehaves, it can be diagnosed and fixed without impacting the other applications
- Applications can be migrated to other servers without impacting their configuration
- Application performance is within 10% of the performance you would see by optimizing all of the components to run within a single address space
- Application tiers have downward dependencies, but a user is not required to access the applications through the top tier (a client can leverage the file server without having to go through the firewall)
- Applications can be tuned for CPU priorities over other applications
- A client must be able to easily shut down one or more of the applications if they choose to not use its features (though reconfiguration of dependencies may be required if the application shut down was pre-configured). The result of the shutdown should be a complete freeing of the resources that the application depended on.
Assumptions
- The user is able to adjust static IP addresses to fit into their network
- Our server can consume as many IP addresses as needed
- A "complete" participation infrastructure consists of a Firewall, a Web Tier (that includes a Web Server with a Wiki Application deployed into it), a Database Tier for storing of Wiki Content, a File Server where the Database stores any files it relies on.
- The application groups that may be "de" consolidated in the future are: Firewall, Web Tier, Database, File Server.
Out of Scope
- Backup of the server content is covered by one of the other Blue Prints, like the [File System Backup to Storage Utility] Blue Print
- High-availability for a node or controller going down is not covered in this Blue Print
Architecture Overview
Virtualization is often used for consolidation of servers and provides the basis for our forward planning. Rather than consolidating existing servers, we will build a single server with applications partitioned and ready to "de" consolidate should we require an upgrade to one of the application spaces due to changes in load.
The following image shows the tiers of applications as well as the stacking of these tiers on top of an opearting system that supports virtualization, such as Sun's Solaris Operating System. It also shows the operating system running on an appropriate server (the same server we will use for the first implementation of the architecture).
Each application will have its own:
- Namespace
- Resources
- IP Addresses
- Packages
- "Local" Storage and Quotas
- Security Management
Implementation Guide - Sun Fire X4500 + MediaWiki + CoolStack
Overview
We are going to use the Solaris Containers technology as the implementation vehicle to fulfill the needs of partitioning the tiers of application deployment. Using Solaris Containers we can derive performance benefits beyond many virtualization technologies as well as simplicity while at the same time maintaining separately manageable application partitions.
The _Bill of Materials_ contains links to specific implementation technologies. The coarse partitioning of
the applications use in this guide are shown below.
The _Implementation Section_ contains details of resource partitioning and exact configuration details.
Each application grouping exists in its own zone. This separation provides isolation of the individual application suites as well as manageable units that can be migrated if de-consolidation is required in the future.
Assumptions
Bill of Materials
Implementation Steps
The implementation steps start with the assumption that Solaris 10 11/06 is already installed and available and the Solaris IP Filter implementation is a part of that installation. I am also assuming that the additional software does not reside in the global zone and will be added to each zone only as necessary. This will create hard partitions of software between the zones and allow each zone to be as much of its own "appliance" as possible. This will help facilitate porting of the zones as well as helping to prevent the manager of one zone from duplicating the purpose of another zone.
This section is a bit different from how one may intuitively think it should be structured. Notably, the _Configuration from Global Zone_ section contains the creation and initial set up of all of the Local Zones. Each identified local zone then has its own section for configuration that occurs from within the local zone.
_NOTE, THE FOLLOWING INSTRUCTIONS HAVE NOT BEEN TESTED, THEY ARE IN PLANNING_
Configuration from Global Zone
The goal of this section is to configure all of the zones with their initial configuration and setup. The individual sections will then boot the zone and do additional setup and configuration beyond that of what should have occurred from the global zone. The goals of this section are to:
- Configure 4 Local Zones: firewall, web, database, nfs
- Each zone will have its own IP Address
- Each zone will have a reasonable amount of storage allocated to it for its individual needs
- Each zone will have a reasonable set of resources allocated to it based on our assumptions of use
On an x4500, the default installation configures the physical ethernet ports to be e1000g0, e1000g1, e1000g2 and e1000g3. The global zone will use e1000g0, the remaining zones will use the remaining ports with the database zone and nfs zone sharing a physical port (e1000g3).
The only zone with a large amount of storage requirements is the nfs zone. All of our database zone data will be stored on an NFS share out of the nfs zone. There will be small requirements for the other zones in terms of locally installed product but not much more. With the large amount of storage on a Sun Fire X4500, we will allocate 10 Gb per zone with the nfs zone receiving, initially, 100 Gb.
As far as CPUs go, the various zones appear to have relatively equal processing needs that fall into the category of receiving a request and, in turn, fulfilling it. The way the applications are organized, the request will move from zone to zone and so ripple across the zones. The only way a particular zone could see unbalanced CPU requirements is if the applications become used for other purposes. This could easily be the case with the nfs zone and may be the case with the web zone. Rather than pre-determining the CPU allocations, we will provide instructions for configuring CPU resources but not constrain them ahead of time.
_Global Zone General Configuration_
Assumptions
- Ethernet - At this point, hopefully the e1000g0 interface is configured and you are able to rlogin or telnet into the system. From here on out, I will assume that the IP Address of the global zone is 192.168.1.100
- ZFS - Based on the default configuration of a Sun Fire X4500, I will assume there is one large storage pool created across the non-boot disks, this pool is labeled 'zpool1' for the remainder of this document.
We will create datasets for each of our zones. During zone configuration, these will be allocated to the respective zones for storage. We also have to create space on the X4500 boot drives within the UFS directories for zone configuration and boot.
Here are the commands, you can adjust as necessary:
''Create boot locations''
- Determine if you have a /export/zones directory, if not, make it
- Make an initial boot directory for each zone
- mkdir /export/zones/firewall ; chmod 700 /export/zones/firewall
- mkdir /export/zones/web ; chmod 700 /export/zones/web
- mkdir /export/zones/database; chmod 700 /export/zones/database
- mkdir /export/zones/nfs ; chmod 700 /export/zones/nfs
''Create ZFS datasets for zones''
- Create a file system for each zone, set initial quotas
- zfs create zpool1/firewall ; zfs set quota=5G zpool1/firewall
- zfs create zpool1/web ; zfs set quota=5G zpool1/web
- zfs create zpool1/database; zfs set quota=200G zpool1/database
- zfs create zpool1/nfs ; zfs set quota=1000G zpool1/nfs
That is all we have to do for general configuration. The network interfaces are allocated during zone configuration.
_Initial Zone Creation_
Each zone will be covered separately in this section, though they are very similar in the configuration.
- zonecfg -z firewall
- create
- set zonepath=/export/zones/firewall
- set autoboot=true
- add net
- set physical=e1000g1
- set address=192.168.1.101
- end
- add dataset
- set name=zpool1/firewall
- end
- add attr
- set name=comment
- set type=string
- set value="Zone containing system firewall"
- end
- verify
- commit
- exit
- zonecfg -z web
- create
- set zonepath=/export/zones/web
- set autoboot=true
- add net
- set physical=e1000g2
- set address=192.168.1.102
- end
- add dataset
- set name=zpool1/web
- end
- add attr
- set name=comment
- set type=string
- set value="Zone containing web infrastructure and mediawiki"
- end
- verify
- commit
- exit
- zonecfg -z database
- create
- set zonepath=/export/zones/database
- set autoboot=true
- add net
- set physical=e1000g3
- set address=192.168.1.103
- end
- add dataset
- set name=zpool1/database
- end
- add attr
- set name=comment
- set type=string
- set value="Zone containing database"
- end
- verify
- commit
- exit
- zonecfg -z nfs
- create
- set zonepath=/export/zones/nfs
- set autoboot=true
- add net
- set physical=e1000g3
- set address=192.168.1.104
- end
- add dataset
- set name=zpool1/nfs
- end
- add attr
- set name=comment
- set type=string
- set value="Zone containing exported filesystems"
- end
- verify
- commit
- exit
That is about it! After zone creation, you have to verigy, install and change the states appropriately. To complete these tasks, you can run the following:
- zoneadm -z firewall install ; zoneadm -z firewall ready ; zoneadm -z firewall boot
- zoneadm -z web install ; zoneadm -z web ready ; zoneadm -z web boot
- zoneadm -z database install ; zoneadm -z database ready ; zoneadm -z database boot
- zoneadm -z nfs install ; zoneadm -z nfs ready ; zoneadm -z nfs boot
From this point, we will log into each zone successfully and continue package setup and leveraging our expansive ZFS file systems and pools that the zones have access to.
Firewall Zone
TBD
Web Zone
TBD
Database Zone
TBD
NFS Zone
TBD
Issues with implementation
- Add physical port redundancy so each zone can fail over to a different physical ethernet port
Guidance for Tuning
TBD
Guidance for "Deconsolidation"
TBD