h1. Architecture Example: Deploying Communications Suite at Sun Microsystems
Robert Chien, July 2008
_(Robert Chien is a member of the Sun IT team and has been designing and building messaging solutions for Sun for the past seven years.)_
This document contains the following sections:
{toc:minLevel=2|maxLevel=2}
h2. What Is an Edge Architecture?
Sun Microsystems' deployment of Communications Suite uses an _edge_ logical architecture. The edge logical architecture adds security for remote access to the two-tiered logical architecture. An edge deployment grants access to a remote, mobile workforce over the public Internet by using only name and password authentication. As messages travel to and from the corporate network over the public Internet, they are encrypted through the use of SSL. No virtual private network is involved. The internal side of the communications transmission is "in the clear" for maximum performance. Access is contained on the "edge" of the deployment, protecting the data stores from unauthorized intrusion.
Business reasons for an edge deployment include:
* Your workforce consists of mobile, remote workers.
* You do not want to install and maintain Communications Suite servers at every remote site.
The following figure represents the edge logical architecture.
*Figure 1 Edge Architecture*
!Communications Suite Attachments^architecture5.gif|alt text="This figure represents the edge logical architecture."!
In the preceding figure, the data stores are located in Tier 2, which is a secure, private network, connected only to the "edge" and "internal" front-end servers. External clients must connect to front-end servers using SSL. Internal clients may use SSL to connect, as the assumption is made that internal access is inherently secure.
h3. Edge Architecture Design Recommendations
* Capacity planning for the Edge tier is difficult to generalize. You should work with the hardware and software vendors who are supplying equipment for your deployment to develop a capacity plan. Nevertheless, you should implement the Realtime Blackhole List (RBL) at your site at the edge tier. The RBL is a list of IP addresses whose owners refuse to stop the proliferation of spam.
* Design the edge tier for minimal latency (less that one millisecond through the entire edge tier).
* Use load balancing algorithms that are load-aware by CPU utilization or by the number of active connections. Round-robin is not an acceptable load-balancing model. With the exception of MTAs (stateless), use sticky-bit load balancing.
* Webmail clients need load balancers that can manage sticky bits, because Webmail interfaces do not share state across Webmail servers.
h2. Edge Architecture Implementation at Sun Microsystems
Sun's edge deployment consists of four global edge presences. Each edge presence serves a community of about 7,000 to 13,000 users. Each presence contains a block of Communications Services, including mail, calendar, instant messaging, and address book.
The following figure depicts the logical edge architecture used at Sun:
*Figure 2 Sun Logical Edge Deployment Architecture*
!Communications Suite Attachments^deploy-arch.jpg|alt text=" This figure shows the logical deployment architecture."!
In the preceding figure:
* Users, whether accessing the deployment from the Internet or intranet, are transferred by load balancers to a global edge's front-end service (indicated by "fe").
* The front-end service then connects to that global edge's back-end service (indicated by "be").
* Both front- and back-end services contact the LDAP directory server (ds) as necessary.
* Note that {{GEO4}} serves a smaller population of users, thus it is smaller in scale.
h2. Data Flows
The following figure shows the flow of data for all the services installed in the edge deployment.
*Figure 3 Data Flow*
!Communications Suite Attachments^data-flow-port-openings.jpg|alt text="This figure shows the flow of Communications Suite data through the deployment."!
In the preceding figure, the lines represent traffic flow and firewall port openings. The Sun Microsystems Communications Suite deployment is architected in a way such that any front end can talk to any back end in any GEO to enable greater flexibility and availability.
{info:title=Note}
The preceding figure shows the planned-for deployment of Sun Convergence.
{info}
h2. Software and Hardware Deployed
The following software and hardware is used in this deployment:
* Software
** Messaging Server 6.2p8.04, 32-bit
** Calendar Server 6.2
** Instant Messaging 7.1
** Directory Server 5.2p3
* Hardware
** 48 front-end Sun Fire T2000 servers (32 GBytes main memory) providing access layer functionality (MTA, MMP, MEM, cshttpd, and Instant Messaging multiplexor)
** 32 back-end hosts:
*** 17 mail clusters, active-passive (Sun Fire V490, 32 Gbytes main memory/4xUS-IV+)
*** 13 calendar stores (Sun Fire T2000)
*** 1 Instant Messaging (Sun Fire V440)
*** 14 Directory Server (4 MMRs, 10 replicas, Sun Fire V240)
** 3 HDS 9990s (RAID 5 LUNs)
h2. Tunings Used and Their Significance
The following tunings are used for Messaging Server IMAP processes:
* {{service.imap.numprocesses = 36}}
* {{service.imap.maxsessions = 225}}
* {{service imap.maxthreads = 250}}
This set of settings is a workaround to avoid the a "Not enough space I/O error" as Sun has many very large folders with tens (sometimes even hundreds) of thousands of messages. These settings enable the support of a total concurrency of 8,100 IMAP connections, which is adequate for the back ends each with approximately 3,000 users. Once the upgrade to Messaging Server 64-bit is complete, these settings will be re-tune to optimize memory usage.
h2. Some Interesting Statistics
To date, Sun is using 72 of 103 Terrabytes of disk storage for its Communications Suite deployment. That works out to about 1.6 Gbytes per employee. In addition, Sun is observing approximately 20 percent year-to-year growth in email storage.
h2. Mobile Communications
Sun provides mobile communications to its employees through a variety of means, including:
* Synchronica SyncML (Calendar and contacts)
* NotifyLink (Blackberry email)
* A corporate address book ("edgebook")
h2. Supported Clients
The Sun deployment supports the following communications clients:
* Thunderbird (the default since December 2007)
* Pidgin
* Lightning
h2. Planned Updates
Sun plans on upgrading to the Communications Suite 6 version during the first half of fiscal year 2009. Items scheduled for implementation include:
* [IMAP IDLE|http://docs.sun.com/app/docs/doc/819-4428/gdqph?a=view], 64-bit Messaging Server, and [very large mailboxes|Administering Very Large Mailboxes]
* DB upgrade and astore for Calendar Server
* LDAP failover and [server pooling|http://docs.sun.com/app/docs/doc/819-4412/gblgs?a=view] for Instant Messaging
* [Sun Convergence|Overview of Sun Convergence]
Robert Chien, July 2008
_(Robert Chien is a member of the Sun IT team and has been designing and building messaging solutions for Sun for the past seven years.)_
This document contains the following sections:
{toc:minLevel=2|maxLevel=2}
h2. What Is an Edge Architecture?
Sun Microsystems' deployment of Communications Suite uses an _edge_ logical architecture. The edge logical architecture adds security for remote access to the two-tiered logical architecture. An edge deployment grants access to a remote, mobile workforce over the public Internet by using only name and password authentication. As messages travel to and from the corporate network over the public Internet, they are encrypted through the use of SSL. No virtual private network is involved. The internal side of the communications transmission is "in the clear" for maximum performance. Access is contained on the "edge" of the deployment, protecting the data stores from unauthorized intrusion.
Business reasons for an edge deployment include:
* Your workforce consists of mobile, remote workers.
* You do not want to install and maintain Communications Suite servers at every remote site.
The following figure represents the edge logical architecture.
*Figure 1 Edge Architecture*
!Communications Suite Attachments^architecture5.gif|alt text="This figure represents the edge logical architecture."!
In the preceding figure, the data stores are located in Tier 2, which is a secure, private network, connected only to the "edge" and "internal" front-end servers. External clients must connect to front-end servers using SSL. Internal clients may use SSL to connect, as the assumption is made that internal access is inherently secure.
h3. Edge Architecture Design Recommendations
* Capacity planning for the Edge tier is difficult to generalize. You should work with the hardware and software vendors who are supplying equipment for your deployment to develop a capacity plan. Nevertheless, you should implement the Realtime Blackhole List (RBL) at your site at the edge tier. The RBL is a list of IP addresses whose owners refuse to stop the proliferation of spam.
* Design the edge tier for minimal latency (less that one millisecond through the entire edge tier).
* Use load balancing algorithms that are load-aware by CPU utilization or by the number of active connections. Round-robin is not an acceptable load-balancing model. With the exception of MTAs (stateless), use sticky-bit load balancing.
* Webmail clients need load balancers that can manage sticky bits, because Webmail interfaces do not share state across Webmail servers.
h2. Edge Architecture Implementation at Sun Microsystems
Sun's edge deployment consists of four global edge presences. Each edge presence serves a community of about 7,000 to 13,000 users. Each presence contains a block of Communications Services, including mail, calendar, instant messaging, and address book.
The following figure depicts the logical edge architecture used at Sun:
*Figure 2 Sun Logical Edge Deployment Architecture*
!Communications Suite Attachments^deploy-arch.jpg|alt text=" This figure shows the logical deployment architecture."!
In the preceding figure:
* Users, whether accessing the deployment from the Internet or intranet, are transferred by load balancers to a global edge's front-end service (indicated by "fe").
* The front-end service then connects to that global edge's back-end service (indicated by "be").
* Both front- and back-end services contact the LDAP directory server (ds) as necessary.
* Note that {{GEO4}} serves a smaller population of users, thus it is smaller in scale.
h2. Data Flows
The following figure shows the flow of data for all the services installed in the edge deployment.
*Figure 3 Data Flow*
!Communications Suite Attachments^data-flow-port-openings.jpg|alt text="This figure shows the flow of Communications Suite data through the deployment."!
In the preceding figure, the lines represent traffic flow and firewall port openings. The Sun Microsystems Communications Suite deployment is architected in a way such that any front end can talk to any back end in any GEO to enable greater flexibility and availability.
{info:title=Note}
The preceding figure shows the planned-for deployment of Sun Convergence.
{info}
h2. Software and Hardware Deployed
The following software and hardware is used in this deployment:
* Software
** Messaging Server 6.2p8.04, 32-bit
** Calendar Server 6.2
** Instant Messaging 7.1
** Directory Server 5.2p3
* Hardware
** 48 front-end Sun Fire T2000 servers (32 GBytes main memory) providing access layer functionality (MTA, MMP, MEM, cshttpd, and Instant Messaging multiplexor)
** 32 back-end hosts:
*** 17 mail clusters, active-passive (Sun Fire V490, 32 Gbytes main memory/4xUS-IV+)
*** 13 calendar stores (Sun Fire T2000)
*** 1 Instant Messaging (Sun Fire V440)
*** 14 Directory Server (4 MMRs, 10 replicas, Sun Fire V240)
** 3 HDS 9990s (RAID 5 LUNs)
h2. Tunings Used and Their Significance
The following tunings are used for Messaging Server IMAP processes:
* {{service.imap.numprocesses = 36}}
* {{service.imap.maxsessions = 225}}
* {{service imap.maxthreads = 250}}
This set of settings is a workaround to avoid the a "Not enough space I/O error" as Sun has many very large folders with tens (sometimes even hundreds) of thousands of messages. These settings enable the support of a total concurrency of 8,100 IMAP connections, which is adequate for the back ends each with approximately 3,000 users. Once the upgrade to Messaging Server 64-bit is complete, these settings will be re-tune to optimize memory usage.
h2. Some Interesting Statistics
To date, Sun is using 72 of 103 Terrabytes of disk storage for its Communications Suite deployment. That works out to about 1.6 Gbytes per employee. In addition, Sun is observing approximately 20 percent year-to-year growth in email storage.
h2. Mobile Communications
Sun provides mobile communications to its employees through a variety of means, including:
* Synchronica SyncML (Calendar and contacts)
* NotifyLink (Blackberry email)
* A corporate address book ("edgebook")
h2. Supported Clients
The Sun deployment supports the following communications clients:
* Thunderbird (the default since December 2007)
* Pidgin
* Lightning
h2. Planned Updates
Sun plans on upgrading to the Communications Suite 6 version during the first half of fiscal year 2009. Items scheduled for implementation include:
* [IMAP IDLE|http://docs.sun.com/app/docs/doc/819-4428/gdqph?a=view], 64-bit Messaging Server, and [very large mailboxes|Administering Very Large Mailboxes]
* DB upgrade and astore for Calendar Server
* LDAP failover and [server pooling|http://docs.sun.com/app/docs/doc/819-4412/gblgs?a=view] for Instant Messaging
* [Sun Convergence|Overview of Sun Convergence]