Architecture Example: Deploying Communications Suite at Sun Microsystems
Robert Chien, July 2008
(Robert Chien is a member of the Sun IT team and has been designing and building messaging solutions for Sun for the past seven years.)
This document contains the following sections:
What Is an Edge Architecture?
Sun Microsystems' deployment of Communications Suite uses an edge logical architecture. The edge logical architecture adds security for remote access to the two-tiered logical architecture. An edge deployment grants access to a remote, mobile workforce over the public Internet by using only name and password authentication. As messages travel to and from the corporate network over the public Internet, they are encrypted through the use of SSL. No virtual private network is involved. The internal side of the communications transmission is "in the clear" for maximum performance. Access is contained on the "edge" of the deployment, protecting the data stores from unauthorized intrusion.
Business reasons for an edge deployment include:
- Your workforce consists of mobile, remote workers.
- You do not want to install and maintain Communications Suite servers at every remote site.
The following figure represents the edge logical architecture.
Figure 1 Edge Architecture
In the preceding figure, the data stores are located in Tier 2, which is a secure, private network, connected only to the "edge" and "internal" front-end servers. External clients must connect to front-end servers using SSL. Internal clients may use SSL to connect, as the assumption is made that internal access is inherently secure.
Edge Architecture Design Recommendations
- Capacity planning for the Edge tier is difficult to generalize. You should work with the hardware and software vendors who are supplying equipment for your deployment to develop a capacity plan. Nevertheless, you should implement the Realtime Blackhole List (RBL) at your site at the edge tier. The RBL is a list of IP addresses whose owners refuse to stop the proliferation of spam.
- Design the edge tier for minimal latency (less that one millisecond through the entire edge tier).
- Use load balancing algorithms that are load-aware by CPU utilization or by the number of active connections. Round-robin is not an acceptable load-balancing model. With the exception of MTAs (stateless), use sticky-bit load balancing.
- Webmail clients need load balancers that can manage sticky bits, because Webmail interfaces do not share state across Webmail servers.
Edge Architecture Implementation at Sun Microsystems
Sun's edge deployment consists of four global edge presences. Each edge presence serves a community of about 7,000 to 13,000 users. Each presence contains a block of Communications Services, including mail, calendar, instant messaging, and address book.
The following figure depicts the logical edge architecture used at Sun:
Figure 2 Sun Logical Edge Deployment Architecture
In the preceding figure:
- Users, whether accessing the deployment from the Internet or intranet, are transferred by load balancers to a global edge's front-end service (indicated by "fe").
- The front-end service then connects to that global edge's back-end service (indicated by "be").
- Both front- and back-end services contact the LDAP directory server (ds) as necessary.
- Note that GEO4 serves a smaller population of users, thus it is smaller in scale.
Data Flows
The following figure shows the flow of data for all the services installed in the edge deployment.
Figure 3 Data Flow
In the preceding figure, the lines represent traffic flow and firewall port openings. The Sun Microsystems Communications Suite deployment is architected in a way such that any front end can talk to any back end in any GEO to enable greater flexibility and availability.
 | Note The preceding figure shows the planned-for deployment of Sun Convergence. |
Software and Hardware Deployed
The following software and hardware is used in this deployment:
- Software
- Messaging Server 6.2p8.04, 32-bit
- Calendar Server 6.2
- Instant Messaging 7.1
- Directory Server 5.2p3
- Hardware
- 48 front-end Sun Fire T2000 servers (32 GBytes main memory) providing access layer functionality (MTA, MMP, MEM, cshttpd, and Instant Messaging multiplexor)
- 32 back-end hosts:
- 17 mail clusters, active-passive (Sun Fire V490, 32 Gbytes main memory/4xUS-IV+)
- 13 calendar stores (Sun Fire T2000)
- 1 Instant Messaging (Sun Fire V440)
- 14 Directory Server (4 MMRs, 10 replicas, Sun Fire V240)
- 3 HDS 9990s (RAID 5 LUNs)
Tunings Used and Their Significance
The following tunings are used for Messaging Server IMAP processes:
- service.imap.numprocesses = 36
- service.imap.maxsessions = 225
- service imap.maxthreads = 250
This set of settings is a workaround to avoid the a "Not enough space I/O error" as Sun has many very large folders with tens (sometimes even hundreds) of thousands of messages. These settings enable the support of a total concurrency of 8,100 IMAP connections, which is adequate for the back ends each with approximately 3,000 users. Once the upgrade to Messaging Server 64-bit is complete, these settings will be re-tune to optimize memory usage.
Some Interesting Statistics
To date, Sun is using 72 of 103 Terrabytes of disk storage for its Communications Suite deployment. That works out to about 1.6 Gbytes per employee. In addition, Sun is observing approximately 20 percent year-to-year growth in email storage.
Mobile Communications
Sun provides mobile communications to its employees through a variety of means, including:
- Synchronica SyncML (Calendar and contacts)
- NotifyLink (Blackberry email)
- A corporate address book ("edgebook")
Supported Clients
The Sun deployment supports the following communications clients:
- Thunderbird (the default since December 2007)
- Pidgin
- Lightning
Planned Updates
Sun plans on upgrading to the Communications Suite 6 version during the first half of fiscal year 2009. Items scheduled for implementation include:
- IMAP IDLE, 64-bit Messaging Server, and very large mailboxes
- DB upgrade and astore for Calendar Server
- LDAP failover and server pooling for Instant Messaging
- Sun Convergence
Comments (8)
Jul 31, 2008
Christian.Knothe says:
Robert, great overview. How do I access the edgebook? The former books.sun.net ...Robert,
great overview. How do I access the edgebook? The former books.sun.net seems to be dead and book-emea.sun.com asks for authorization but doesn't show any results.
Thanks,
Christian
Jul 31, 2008
rsc says:
Christian, the access instructions (and more) can be found here: [TechSoluti...Christian, the access instructions (and more) can be found here: [TechSolutions:book.sun.com].
Jul 31, 2008
Christian.Knothe says:
Brilliant, thanks a lot.Brilliant, thanks a lot.
Sep 29
brian_j_murrell says:
I don't see anything at that book.sun.com (the wikis page the URL points to) abo...I don't see anything at that book.sun.com (the wikis page the URL points to) about edgebook configuration. Has it moved or be removed, or are there any other pointers about it?
Followup: I did find a reference in the internal wiki which seems to have what I needed.
Aug 14, 2008
uqbar says:
"edgebook" sound interesting - for those of us who are not Sun employees, can yo..."edgebook" sound interesting - for those of us who are not Sun employees, can you tell us more about this?
P.S.
This is not just idle curiosity; we have a messaging installation similar to Sun's (smaller in scale!), and we're always looking for ways to make improvements.
Aug 14, 2008
rsc says:
uqbar, edgebook is an LDAP-based directory lookup service we provide on the Edge...uqbar, edgebook is an LDAP-based directory lookup service we provide on the Edge that enables Sun employees to do name search and email address completion from LDAP-enabled clients such as Thunderbird, Apple's Mail.app and Outlook. The service requires authentication and encryption, and only returns a filtered list of attributes. It complements our email and calendar services on the Edge really well, and our users love it!
Dec 10, 2008
JimKlimov says:
And how did you implement that? Our users would love that too Is it some read-...And how did you implement that? Our users would love that too
Is it some read-only Directory Proxy Server data view with ACI's
(I clicked around a test DPS but didn't quickly find any ACI's,
nor a means to assign the data view to specific clients/subnets)?
Or is it a Directory Server consumer with ACI's?
Which attributes proved to be required-and-adequate for address
searches and completion, without compomising security?
Thanks for the great informative article,
Jim
Dec 11, 2008
rsc says:
Jim, it's a Directory Proxy server (DPS) with a network group set up to allow re...Jim, it's a Directory Proxy server (DPS) with a network group set up to allow read access to certain attributes. DPS relays queries it receives to the Directory server consumer.
There are no ACIs on the Directory server consumer underneath. The access control is at the DPS level.
Here is the doc on network groups: http://docs.sun.com/source/816-6391-10/ic_groups.html#11284
The attributes include employees email address and name. The list was reviewed by our security and privacy officers.
Hope that helps.