Understanding a SAP SD 2-tier benchmark result

!Updated to new benchmark rules 2009!

SAP introduced a couple of benchmarks to help their customers find the appropriate hardware for their SAP systems. The most prominent benchmark is the SAP SD 2-tier benchmark. Unfortunately, there is more than only one variant of this benchmark, so the interpretation of the results can be complicated. This article tries to give you a better insight to the benchmark to let you understand and compare its results.

The SAP SD 2-tier benchmark simulates user interaction in the SD (sales and distribution) module: A users logs on, creates a customer order, fills it with five line items, saves it, creates a delivery, displays the order, and so on. The complete description of the benchmark can be found here: http://www.sap.com/solutions/benchmark/sd.epx

When browsing the results of the SAP SD 2-tier benchmark (http://www.sap.com/solutions/benchmark/sd2tier.epx), there are quite a few criteria mentioned that characterize a benchmark result. All these criteria potentially influence the result, so I try to explain all of them to give you an overview.

The first two columns on this page give information about the company submitting the benchmark:

• The "Date of Certification" is the date, when SAP accepted the benchmark result and published it.
• The "Technology Partner" is the company, that is responsible for the benchmark and sells the used server.

The next columns show key figures of the benchmark:

• The "Number of Benchmark Users" is the key result of the SAP SD benchmark. This number shows how many simulated users were simultaneously running their load on the machine.
• The "Average Dialog Response Time (sec)" is the time the system needed to fulfill a users request. The maximum response time accepted by SAP is 2 seconds today, in 2009 it will be lowered to 1 second. So most Technology Partners try to get close to 2 seconds, because shorter response time means less simultaneous users.
• The "Dialog Steps Per Hour" closely correlates to the "Number of Benchmark Users" and the "Average Dialog Response Time (sec)": Every Benchmark User perforce the same number of Dialog Steps during th benchmark run.
• The "SAPS" value is derived from the Dialog Steps and is one of the most cited results of a SD benchmark. 100 SAPS are 2.000 fully processed order line items per hour. In technical terms this means for the SD benchmark: 6.000 dialog steps are 100 SAPS (more information can be found here: http://www.sap.com/solutions/benchmark/measuring/index.epx). The result in SAPS can be used for sizing purposes: SAP quicksizer results are measured in SAPS, so the SAPS is the currency to compare SAP systems and SAP environments.

The last columns are very important to interpret the result, because there are many influences to the result. If you want to compare two or more SAP SD benchmark results, you should first compare these columns:

• The "Operating System" is self-explanatory, it's the OS running on the machine that is benchmarked. Of course, the OS has an influence on the result. A high-performance OS will cause better results than a OS, that is not dedicated to run application servers with many simultaneous requests and much throughput. Unfortunately, there is no matrix containing "OS Factors" to convert a Solaris result into a Windows result or vice versa. Only by comparing benchmarks running on (mostly) identical hardware, the same release and the same RDBMS, you can figure out an os factor for your comparison.
• The "RDBMS Release" specifies which database was used. On the first look, the database seams to play a important part on overall performance. But a closer look shows the impact of the database is not such big: Compared to the Response Time the average processing time on the database is quite small. Only 5-10% of the overall CPU time of a request in the benchmark is database related. So the impact of the database is not such important for the overall result.
• The "ERP Release" specifies the release of SAP ERP used in the benchmark. Currently, the ERP Release valid for the benchmark is "SAP enhancement package 4 for SAP ERP 6.0 (Unicode)", sometimes abbreviated as "EHP 4". There is a small but high impacting annex in brackets: (Unicode). Starting with this release, using Unicode became mandatory, in earlier releases you have to pay attention on the this little word: It means the application server and the database are using unicode. In technical terms: the SAP Application Server is coding all data in memory using UTF-16, and database uses UTF-8 or UTF-16 (depending on the database). So using unicode leads into a higher memory demand (40%-50%) and higher CPU load (10%-30%) For more information see this public available presentation or this SAP-Note. Because the SD benchmark is depending on CPU, memory and I/O, comparing unicode results with "non-unicode" always means to convert the unicode to non-unicode results or vice versa.
  Of course there can be older benchmarks using older ERP releases like ERP 6.0 (without Enhancement Packages) 2004 (64-bit), ECC 5.0 or even older releases like 2.2 or 3.1C. This impacts the result, older releases needed less compute power to deliver the same amount of Dialog Steps, so you have to convert the results of older benchmarks to make them comparable.
• The "Central Server Additional Specifications" give the Specifications of the used hardware. The format is standardized to allow faster comparisons: Model name, number of processors/cores/threads, Processor name, frequency and cache specifications. Of course, this has a strong on the result. The used server must become general available within 180 days after certification.
• The Column "Central Server Memory" states the amount of memory used for the benchmark.

By following the link in the last column you can display or download the certificate in PDF format.

It is not easy to compare two results. To make valid and fair comparisons, it's important to consider the most impacting factors shown above.

• First of all, you should assure to compare the same release (ERP 5.0, ERP 6.0. ERP 6.0 unicode...). If the releases are not identical, you can convert the results using SAP's factors for unicode/non-unicode migrations or their recommendations for system upgrades. For example: add 15% to all unicode results to make them comparable to non-unicode results.
• Then you should be aware of the impact of the Operating Systems. If there are different OS, try to find results using the same hardware but different OS to get a sense of the impact.
• The impact of the database is quite low, so you can ignore it.

After including all these factors, you can compare the results.

Sometimes you'll find benchmarks with unexpected low results. These benchmarks are so called certification benchmarks. The Technology Partner only submitted this result to get a certification for this particular server. Of course, those benchmarks can not be used for comparisons.

Very unusual are benchmarks using virtualization technology. Technology partners want to max out their results, so overhead created by any visualization layer is unwanted and avoided. Of course there are at least a few benchmarks that can be used to get a sense of the impact of a visualization layer. For example Sun ran two benchmarks using a T2000 without visualizationand with four Solaris Container (Certificates 2006029, 2006030, 2006031, 2006032).