The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
© 2005 Microsoft Corporation. All rights reserved.
Microsoft, the .NET logo, Visual Studio, Win32, Windows, and Windows Server 2003 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
Microsoft Corporation • One Microsoft Way • Redmond, WA 98052-6399 • USA
The goal of this paper is to present a comparison of Web application servers: the Microsoft .NET Framework 2.0 running on Windows Server 2003, and the latest release of IBM WebSphere Application Server, version 6.0.2.3. According to a recent IDC study from October of 2005, roughly 78% of mission critical application projects under development worldwide use an application server. In this study, IDC also found that the combination of .NET and Windows Server 2003 is now the most commonly used application server for mission critical applications, exceeding IBM WebSphere usage with 37% usage for .NET/Windows Server vs. 12% usage for IBM WebSphere.
This paper presents the results of a new benchmark that compares the performance of those two application servers. The benchmark is based on the PlantsByWebSphere sample application, the primary sample application included with WebSphere 6. For this benchmark, we implemented a functionally equivalent .NET application (DotNetGardens), and also made several performance optimizations to the IBM application code base to ensure a fair comparison of performance for data-driven Web applications.
See IDC Research 2005 Mission Critical Survey (Oct 2005) at http://msdn.microsoft.com/vstudio/java/compare.
The complete source code, all test scripts and all testing methodology for this benchmark are available online. Any reader may download and view the actual code for all implementations tested, and may further perform the benchmark for themselves to verify the results. The benchmark kit can be downloaded from http://msdn.microsoft.com/vstudio/java/compare/.
The following shows the complete matrix of the implementations tested (all are tested on the exact same hardware). In the future, we also plan to publish a WebSphere EJB implementation that works with SQL Server 2005 using the new, soon-to-be released Microsoft SQL Server JDBC driver.
|
Windows Server 2003 as App Server OS |
Red Hat Linux as App Server OS |
||
Implementation Name |
Backend Oracle10G DB |
Backend SQL Server 2005 DB |
Backend Oracle10G DB |
Backend SQL Server 2005 DB |
PlantsByWebSphereEJB
|
X |
|
X |
|
PlantsByWebSphereJDBC |
X |
X |
X |
X |
DotNetGardens |
X |
X |
|
|
The results show that the .NET 2.0/Windows Server 2003 implementation of the benchmark application running against SQL Server 2005 outperforms the Java EJB-based WebSphere 6.0.2.3/RedHat Linux implementation by up to 183%. The results also show that .NET achieves significantly better price/performance ratios than WebSphere 6 on all implementations tested. This paper will also show that Java applications can get good performance when connecting to SQL Server 2005: the performance of the PlantsByWebSphere JDBC-based application that uses the soon-to-be-released Microsoft SQL Server 2005 JDBC driver is approximately equivalent to the performance of the equivalent application that connects to Oracle 10G.
In order to conduct the benchmark in a fair and complete manner using an application that conforms to IBM’s best coding practices, we chose the PlantsByWebSphere sample application as the basis for comparison, since IBM developed the Java application. The PlantsByWebSphere sample application is a J2EE-based application created by IBM according to their EJB-based development guidelines, and ships with every copy of WebSphere 6.0 as their primary J2EE sample application. The application uses a very typical J2EE design: Enterprise Java Beans including a stateless session bean façade to wrap entity beans that use container-managed persistence to access the database. The user interface is Web-based, relying on Java Server Pages and Servlets as the core architectural elements of the UI layer. The IBM application uses model objects to represent database information, and these model objects are passed between the EJBs and the UI layer to maintain a clean separation of UI, business and data layers of the application.
In order to provide a comprehensive assessment of the scalability and performance of the two platforms for this benchmark, several optimizations and additions were made to the PlantsByWebSphere sample application as provided by IBM with WebSphere.
We elected to make these changes to better reflect the design a customer would use when actually deploying an application on WebSphere Application Server in a production environment. All of these design changes are supported by the common design guidelines for J2EE applications. Each of these changes independently resulted in better performance of the modified application design, or better, more practical function. The aggregate affect of all of the performance- related changes on the actual performance of the application was significantly positive.
Finally, in addition to those changes, we also constructed a new, alternative EJB-free implementation. There is some disagreement in the J2EE community about whether EJBs should be considered harmful, in terms of overall performance and scalability—as well as application complexity. While some (including IBM) maintain that EJBs are the best architecture for building a data-driven Web application, to paint a complete picture of application performance, we created a second implementation of the PlantsByWebSphere application. This implementation uses no EJBs at all (either session beans or entity beans), but is precisely functionally equivalent to the original IBM EJB implementation and the DotNetGardens application. This implementation instead uses only JDBC logic to access the database. The JDBC logic is encapsulated in a separate data access layer (much like the ADO .NET implementation), activated via the same servlet architecture employed in the original EJB-based PlantsByWebSphere sample application. This second implementation shares much code and design with the original EJB-based implementation: the same model objects are used to pass data between tiers, the same base servlets and UI layer is used, and the other optimizations described above also apply.
While we tested the EJB-based application against only the Oracle 10G backend, we tested this second implementation against Oracle10G as well as SQL Server 2005 using the new and soon-to-be released Microsoft SQL Server JDBC driver. A Beta version of this driver is available on MSDN for free download. The development of a new, optimized (and free) JDBC driver for SQL Server 2000/2005 by Microsoft with full support from Microsoft is an exciting development for J2EE developers that want to use SQL Server 2005 with their J2EE-based applications. We expect to release this driver commercially in January 2006, and the Beta 2 release is available on MSDN.
We are providing on the Web the full source to the optimized EJB implementation as well as the new JDBC/Servlet implementation as tested, in the benchmark kit.
We created a functionally equivalent application to PlantsByWebSphere, called DotNetGardens, in order to compare the performance of the two application server platforms. DotNetGardens operates against the same backend database schema as PlantsByWebSphere. The DotNetGardens application matches the functional behavior of the PlantsByWebSphere application, and employs a similar 3-tier logical architecture that maintains a clean separation of UI, business and data-tier operations. This 3-tier architecture is based on Microsoft-recommended best practices for enterprise development. The DotNetGardens application uses .NET assemblies to encapsulate all data access in a single layer of data-access classes. These data access classes use ADO.NET 2.0 to provide the data access functionality of the application just as the EJB entity beans provide data access for the WebSphere application. The .NET implementation uses no stored procedures, and instead uses pre-prepared SQL statements to match the WebSphere implementation. The data access classes are front-ended by a distinct business tier. Finally, the UI layer of the application was developed using ASP.NET 2.0 Web Forms. Like the WebSphere implementation, all data records are encapsulated in model classes that are passed between tiers to maintain a clean separation of UI, business and data tier operations. The DotNetGardens application will work against both an Oracle10G backend database using the .NET Framework Data Provider for Oracle (which is integrated into the .NET framework itself), and with SQL Server 2005/2000. The DotNetGardens application is also available for download as part of the benchmark kit.
PlantsByWebSphere EJB Implementation
PlantsByWebSphere JDBC Implementation
DotNetGardens .NET Implementation
All implementations were tested on the following application server hardware:
The software installed on this server included the following OS/App Server combinations:
Windows Server 2003 Standard Edition x86 (32-bit)
Red Hat Linux Advanced Server (Release 4) for AMD Opteron
All implementations were tested with the following dedicated database server hardware:
(One RAID array used for core datafiles, one used for logging)
The software installed for the databases included:
The database was closely monitored to ensure that in no case it represented a bottleneck on performance (either in terms of CPU utilization, network utilization, or physical disk utilization). In other words, the benchmark results indicate the top-end performance achieved by the application server for each implementation tested given plenty of database capacity to handle all peak-performance loads. In all benchmark runs, we were able to fully saturate the application server/tested benchmark implementation at above 96% CPU utilization to accurately capture as intended the peak throughput of the application server itself, inclusive of the middle-tier data access technology used by the various implementations. It is important to remember that this is not a database benchmark, since both SQL Server 2005 and Oracle10G could have handled user loads well-beyond those supported by the middle tier hardware/software platforms we tested. Instead the benchmark measures the performance of the middle tier and how fast it is at connecting to and working with various backend databases in a Web-based scenario, using the most commonly employed data access techniques and drivers for each platform.
Mercury LoadRunner 7.51 was used to benchmark the various implementations. The LoadRunner test scripts are published with the source code so that customers can easily test the applications for themselves. 20 physical agents machines were used to generate load, and all agents were set to run with a 1-second think time. Hence, peak TPS rates also roughly equate to the number of concurrent users supported on the tested platform for each implementation tested. The scripts exercised the following functionality in the application:
For the benchmark, the Plants customer database was loaded with 10,000 unique customers, and the inventory database was loaded with 20,000 unique products.
To aid customers in comparing value, and not just raw performance, we priced the software licenses required to support the application servers used in each of the configurations tested (exclusive of the backend database license costs). To build and run the PlantsByWebSphere application as configured for IBM WebSphere Application Server Express v6.0 on Red Hat Linux ES, the total software license cost is $12,054. To build and run the functionally equivalent DotNetGardens application with .NET 2.0 on Windows Server 2003, the software license cost is $6,668.92. The IBM configuration is nearly twice as expensive in middle-tier software license costs.
See the Appendix for detailed breakdown and information sources.
The following chart depicts the Price/Performance ratio. The numbers in this chart are computed by dividing the software license cost for each configuration, by the peak TPS achieved by that configuration. This is a measure of the value of the application server software. For example, an application server that performs twice as well but at double the cost would equate to an equivalent price/performance score. Lower price performance numbers represent greater value (lower cost/better performance). No database license cost is included in the calculations below, it is based solely on the middle-tier software license cost. See the appendix for the detailed cost breakdowns and their sources.
In most cases a valid benchmark result of peak TPS rates should be reported at the point the CPUs of the system under test are “saturated” – in other words, it should be running at or near 100% CPU utilization without over-saturation. When the CPU capacity is saturated, the machine is achieving peak throughput for the given configuration and application because at 100% CPU utilization, there are no other bottlenecks in the system other than the processing efficiency of the system under test . This table shows the CPU levels and peak transaction rates achieved, for each different application implementation that we tested:
Implementation |
Peak TPS |
App Server CPU Utilization Peak |
WAS 6 EJB/ Oracle10G/ Windows Server 2003 |
917 |
98% |
WAS 6 JDBC/ Oracle10G/ Windows Server 2003 |
1835 |
99% |
WAS 6 JDBC/ SQL Server 2005/ Windows Server 2003 |
1805 |
98% |
.NET 2.0/Oracle10G/Windows Server 2003 |
1804 |
96% |
.NET 2.0/SQL Server 2005/Windows Server 2003 |
2915 |
96% |
WAS 6 EJB/Oracle 10G/RH Linux Enterprise Server |
1030 |
98% |
WAS 6 JDBC/Oracle10G/RH Linux Enterprise Server |
2105 |
96% |
WAS 6 JDBC/SQL Server 2005/RH Linux Enterprise Server |
2053 |
96% |
One of the most sensitive tuning areas for many J2EE applications is the java heap size. We tested all IBM implementations with various heap size settings to determine the optimal heap sizes for both the EJB and JDBC-based implementations. For this application we did not see significant performance gains if the heap size was configured beyond 1000 megabytes of RAM, since the application itself has a small memory footprint. We also found that the EJB-based implementation performed at its best with a larger heap size than the JDBC implementation, possibly because of the memory usage for the EJB cache. We did find that increasing the heap size beyond 1800MB for the JDBC implementation caused a slight decrease in performance, probably because of larger overhead associated with garbage collection in a larger heap. In other applications, this threshold will be different.
Note that it is not necessary to optimize the heap size with a web application based on .NET running on Windows Server 2003. While we did not measure the time and labor consumed by performing this optimization on the WebSphere application, it can often represent a significant amount of skilled labor. In addition, this analysis must be performed again, each time the profile of the application (either the source code, or the machine configuration, or both) changes significantly.
The following tables show the data we measured to help us choose the heap size for both the JDBC and EJB implementations. The data presented here is for the RedHat Linux OS. However, we followed the same analysis for WebSphere 6 on the Windows OS. For a complete list of all tuning parameters set for both IBM WebSphere and .NET, please refer to the appendix.
The results of this benchmark show that:
Results in an actual data-driven Web application will vary. Most customer applications are more complex than this benchmark. No benchmark can hope to capture all actual real-world requirements for all customers, and no single benchmark can answer all performance questions. Nevertheless this benchmark is interesting for enterprise developers because the application, while simple, uses and exercises many of the same enterprise architectural building blocks used in complex, real-world applications. We hope this benchmark provides solid evidence that .NET 2.0 performance and price/performance for building and running a data-driven Web application can be superior to IBM WebSphere 6. We encourage customers to download the benchmark kit, modify it according to their own environment, and perform their own tests.
The following table shows the pricing breakdown for the application server middle tier to run the Plants application on a single 4-cpu middle tier server, as done in this benchmark. The pricing compares .NET running on Windows Server 2003 to IBM WebSphere Express running on Red Hat Linux. The pricing is inclusive of operating system license cost for a supported version of each OS, the cost of the application server itself, and one developer seat license for the Microsoft and IBM development tool used to build the applications. Note that database pricing is not included in this chart, please refer to Oracle and SQL Server 2005 pricing comparisons at http://www.microsoft.com/sql if you want to compare database pricing.
|
Microsoft .NET 2.0 vs. IBM WebSphere 6.0 License Cost Comparison As Configured for the Plants Benchmark |
|||
Item |
Microsoft |
Cost |
IBM |
Cost |
Base Server OS |
Windows Server 2003 Standard Edition @ $1089.85 / system |
$1089.85 |
Red Hat Enterprise Linux AS @ 1 year subscription |
$1499.00 |
Application Server Function |
(Included in Windows Server) |
-- |
WebSphere Application Server Express, v6.0 @ $2000/cpu |
$8000.00 |
Unlimited External Connections |
External Connector @ $3033.07/ system |
$3033.07 |
(Included in WAS Express) |
-- |
One Developer seat |
Visual Studio 2005 Professional with MSDN Premium Subscription, 1 annual charge |
$2518.00 |
Rational Application Developer for WebSphere software, floating user, 1 st year |
$2155.00 |
Development-time runtime license |
(Included in MSDN Enterprise subscription) |
-- |
WAS Express Developer User |
$400 |
Media |
CD kit for Windows Server 2003 Standard Edition |
$28.00 |
(included in IBM.com price) |
$-- |
Total |
|
$6668.92 |
|
$12054.00 |
Costs for IBM were obtained from the IBM.com online store, and the Red Hat online store. Red Hat Enterprise Linux AS is required, as opposed to the less expensive Red Hat Enterprise Linux ES, because this configuration uses 4 cpus, and the maximum number of CPUs supported by the Red Hat Enterprise Linux ES license is 2. Windows Server 2003 Standard Edition supports up to 4 CPUs. A single seat license for the RWD Developer tool is included in WAS Express, but the EJB implementation of the Plants app (as originally developed by IBM) requires support for EJBs, and therefore we separately priced Rational Application Developer for WebSphere 6 to meet this requirement. The price of a floating user license for RAD is $2155.00 for the initial year under a current (as of October 2005) promotion; the normal price is $7000/user.
License costs for all Microsoft software were obtained from a Microsoft authorized reseller, except for Visual Studio 2005.
**Note: The pricing for Visual Studio 2005 Professional with MSDN Premium Subscription is estimated retail pricing, not actual reseller pricing. The actual reseller price of this item was not available at the time of printing, but will likely be slightly lower than the estimated price used here.
WAS 6.0.2.3
Session State set to 5 minute expiration (in-process session state) Access Log Turned Off Performance Monitor Infrastructure Turned Off Diagnostic Trace Turned Off System Out Off EJB Cache Size = 20000 HTTP Channel maximum persistent requests = -1 HTTP Channel readTimeout = 6000 HTTP Channel writeTimeout = 6000 HTTP Channel persistentTimeout = 3000 Minimum Web Container threads = 75 Maximum Web Container threads = 75 Minimum ORB threads = 40 Maximum ORB threads = 40 Minimum Default threads = 20 Maximum Default threads = 20 Heap Sizes: 1. 1640 MB EJB Implementation on Windows Server 2003 (this is max for 32-bit WebSphere on Windows) 2. 2300 MB EJB Implementation on Linux (this is max for 32-bit WebSphere on Linux) 3. 1640 MB JDBC Implementation on Windows 4. 1800 MB JDBC Implementation on Linux EJB implementation: Access intent policy set to Optimistic Read Isolation Level for read methods of the Inventory and Customer entity beans
IBM HTTP Server
IBM HTTP Server Windows: Access Log Off Max KeepAlive Requests 3000 2048 Max threads 2048 Threads/child IBM HTTP Server Linux: Access Log Off Max KeepAlive Requests 3000 ThreadLimit 50 ServerLimit 64 StartServers 50 MaxClients 3200 MinSpareThreads 100 MaxSpareThreads 100 Threads/Child 50 MaxRequests/Child 0
Linux Tuning
net.ipv4.tcp_max_syn_backlog=1024 kernel.msgmni=1024 kernel.sem=1000 32000 32 512 fs.file-max=65535 kernel.shmmax =4294967295 net.core.netdev_max_backlog = 20000 net.core.somaxconn = 20000 net.ipv4.tcp_fin_timeout = 30 net.ipv4.tcp_syn_retries = 20 net.ipv4.tcp_synack_retries = 20 net.ipv4.tcp_sack = 0 net.ipv4.tcp_timestamps = 0 net.ipv4.conf.all.arp_ignore = 3 net.ipv4.conf.all.arp_announce = 2 Open File Handle limit (soft) increased to 20000
Windows Server 2003 Tuning
Added registry parameter settings for tcpip service (CurrentControlSet/Services/tcpip/parameters): TcpTimedWaitDelay set to decimal 30 MaxUserPort set to decimal 32768
.NET Worker Process
Rapid Fail Protection off Pinging off Recycle Worker Process off
ASP.NET
Session state timeout set to 5 minutes ((in-process session state) Authentication set to “None” to match anonymous access of IBM WebSphere
IIS 6.0 Virtual Directory
Authentication Basic Only Access Logging Off
Agents were set to: 1. Simulate a new user on each request (each script iteration opens, then at the end closes its own connection to the app server, although http keepalives are enabled in all cases). 2. Download only HTML resources returned by the application server, and not graphics files or any other non-HTML resources.
The PlantsByWebSphere Java application we created for use with an MS SQL Server backend will operate with the Beta2 release of the new Microsoft SQL Server JDBC driver available on MSDN. This driver is based on a completely different codebase from our prior driver for SQL Server 2000 and offers up-to-date features and better performance. However, we tested with a pre-beta2 release of this driver do to a current bug in the Beta 2 release that affects only the new search logic we added to the application (a single query in the app). This bug prevents SQL Server from properly using its query optimizer on LIKE clauses when run via the beta JDBC driver. We have since addressed this bug for the upcoming commercial release. However, to replicate our results or perform further benchmarks on the SQL Server Java application in the benchmark kit, customers can obtain a copy of the pre-beta 2 build we used by emailing benchnet@microsoft.com. We are also working on a PlantsByWebSphere EJB implementation for SQL Server 2005 that will demonstrate the use of this new driver with Container Managed Persistence/EJBs.
Note that we priced the less expensive IBM WebSphere Express even though we tested with the enterprise-recommended IBM WebSphere Network Deployment Edition because technically on a single 4-processor server a customer could use IBM Websphere Express vs. the more expensive Network Deployment Edition and reduce their WebSphere license costs. However, should a customer want to implement the Plants application in a cluster with load balancing and shared session state, they would typically license IBM WebSphere Network Deployment Edition which would cost an additional $13,000 per CPU vs. Express on each server it is deployed to. The .NET implementation supports network load balancing and shared session state for clusters with no additional license cost beyond purchasing additional copies of the operating system for each server in the cluster (which also applies to WebSphere clusters).