June 7, 2005
Comparing Web Service Performance
WS Test 1.1 Benchmark Results for.NET 2.0, .NET 1.1,
Sun One/ JWSDP 1.5 and IBM WebSphere 6.0
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
Ó 2001 Microsoft Corporation. All rights reserved.
Microsoft, the .NET logo, Visual Studio, Win32, Windows, and Windows Server 2003 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
Microsoft Corporation • One Microsoft Way • Redmond, WA 98052-6399 • USA
1297
WSTest is a Web Service benchmark created by Sun Microsystems and augmented by Microsoft. The benchmark tests various web service operations across varying SOAP object sizes. Sun’s original WSTest 1.0 benchmark kit can be downloaded from:
http://java.sun.com/developer/codesamples/webservices.html#Performance
WSTest 1.1 is a Microsoft implementation of WSTest 1.0 that mirrors the Sun implementation of the EchoVoid, EchoStruct and EchoList tests. In addition, it includes one new test, GetOrder, that tests a more complex object type that simulates a purchase order. This report details the performance of .NET Beta2 (build v2.0.50215) vs. NET 1.1, Sun’s latest Java Web Services Developer Pack (JWSDP 1.5) with the SunOne Enterprise HTTP Server 6.1, and IBM Websphere 6.0 with the IBM HTTP Server 6.0. All tests were conducted on 32-bit Windows Server 2003 RTM.
The configurations tested include:
- SUN JWSDP1.5/SUN HTTP Server 6.1
- IBM WebSphere 6.0/IBM HTTP Server 6.0
- .NET 1.1/IIS 6.0
- .NET 2.0/IIS 6.0
Extensive time was taken to properly tune all configurations following vendor best practices and iterative testing to achieve best results for each configuration. In all cases, tests achieved 100% CPU saturation under peak throughput client loads (meaning no significant bottlenecks other than CPU saturation), indicating proper tuning of the middle tier across all products. In all configurations, tracing, logging, authentication and session state tracking is turned off for all products. Http 1.1 KeepAlives are enabled and set to a sufficient number to achieve best results. Java heap sizes and thread settings were tuned to achieve maximum performance. All tests were conducted on the same hardware, an HP DL858 2-processor AMD Opteron @1.8 GHz and 4 MB RAM with gigabit networking.
Sun’s original WSTest 1.0 uses a custom driver program (one for Java, one for .NET) to drive load against the backend Web Services deployed on the server. Their original published results were flawed in that they did not properly configure the MaxConnections setting in the .NET Machine.Config for the .NET version of the benchmark driver program, which essentially throttled the driver to two network connections to the server, preventing full saturation of the server. Corrected results were published by Microsoft shortly thereafter using the java custom driver program to drive load, and can be viewed at:
http://download.microsoft.com/download/1/9/b/19bc8aa7-05fa-4e86-a612-c2cc181e4ee6/sun_ws_benchmark_response.pdf
Sun’s original test methodology used a single client machine to drive load against the server using no think time between client requests. The driver program established a connection to the server across a limited number of threads (16), and the threads re-used this connection without closing/re-establishing connections between requests. This methodology is not ideal, however, because it does not simulate real-world usage of a Web Service. Typically in a real-world scenario, many different clients make requests to the web service, and the server must handle load across many physically distributed clients as they establish new connections to the Web Service. For the results published in this paper, we have used two test methodologies for a more complete picture of Web Service performance. Both use Mercury LoadRunner’s SOAP client to drive load across all products tested. By using Mercury LoadRunner instead of a custom driver program, the tests are more accurate in that all products are tested using the exact same test harness. Furthermore, Mercury LoadRunner is capable of driving load using many distributed client machines, while Sun’s custom driver program was meant to be run from a single client machine.
The two test methodologies employed achieve very different results for the Java-based products tested. In the single-client configuration that mirrors Sun’s original methodology, client threads re-use existing network connections on each iteration, and the client thread does not close out its connection after executing it’s SOAP request. This test methodology does not realistically simulate usage for thousands of clients, each connecting from a unique IP address, executing a request, and then closing it’s connection to the server. Therefore, in addition to this methodology, we also tested the same products using 50 physical client machines, with a 1 second think time before a request, and a connection close at the end of each request. This more realistically simulates real-world usage of a deployed web service, with the server having to handle many more open connections and physical IPs through it’s network infrastructure and keepalive system.
Finally, the test results included in this paper use Sun’s recommended SunOne Enterprise HTTP Server 6.1 instead of TomCat, which was used in Sun’s original tests. Sun recommends the use of this HTTP server in published materials for enterprise scenarios given it’s more advanced management capabilities. Also, we tested the latest IBM WebSphere 6.0 application server, configured to use IBM’s HTTP Server. Both the IBM HTTP Server and the SunOne Enterprise HTTP Server are derived from Apache.
As with other .NET benchmark tests, full source code is published so that customers and other vendors can replicate the tests and verify/comment on the results, or use the tests on other platforms or against other products.
WSTest 1.1 consists of 4 distinct web service methods. The methods isolate the SOAP network stack/serialization performance of each platform tested. The web service calls included in WSTest 1.1 include1:
- EchoVoid – sends and receives an empty message, with no deserialization/serialization
- EchoStruct –receives an array of any length as the input parameter with each element consisting of a structure. The list is then sent back to the client as the return type. The repeating structures within the list contain one element each of type integer, float and string datatypes. The longer the listsize, the more work required in deserialization and re-serializing the passed SOAP object to/from XML.
- EchoList-sends and receives a linked list of any size, each element consists of the same structure used in EchoStruct.
- GetOrder – this function simulates a request for a complete purchase order, taking two integer input parameters with an order object returned from the Web Service over SOAP. The order object is a more complex structure that includes order header information, a customer sub-class with shipping address and billing address structures, as well as any number of lineitem objects. For this test, the web services return exactly 50 lineitems for each request as part of the returned order object.
In the results published in this paper, peak throughput is reported as the number of web service requests handled per second by each product (TPS). Peak throughput is achieved at peak CPU saturation of the server, but not over-saturation. In other words, at peak throughput each product is able to process all incoming requests using available CPU cycles without queuing any requests. In such cases, response times are all in the millisecond range since no requests are being queued. Care was taken in testing to ensure that the peak throughput reported represents the top of the throughput curve for each product as client loads were steadily increased.
HP DL585 with 2 x 1.8 GHZ AMD Opteron Processors
4 GB RAM
Gigabit Networking
Windows Server 2003 Enterprise Edition (32-bit)
In isolation the following software platforms were tested on the above hardware:
- .NET 1.1
- .NET 2.0 Beta2 (v 2.0.50215 available on MSDN)
- JWSDP 1.5 Update 2 with SunOne HTTP Server 6.1
- IBM WebSphere 6.0 with IBM HTTP Server 6.0
50 physical Dell client computers configured with 512 MB RAM and one Intel Celeron CPU @ 500 MHZ, and gigabit networking.
Each client is running Mercury LoadRunner agents, using Mercury’s SOAP driver program. Each client machine is capable of generating many concurrent virtual users (threads), with the overall testbed capabable of generating well over 10,000 concurrent user requests to the backend system. In the more realistic multi-client tests, each virtual user is configured with a one second think time and all 50 clients are used. In the less-realistic single client test, a single client computer is used running 16 threads, with no think time between requests. In both test methodologies, care was taken to ensure servers were operating at full saturation and peak throughput was accurately captured. Because the think time used in the multi-client test is one second, the peak TPS rate reported almost exactly matches the number of virtual users being supported simultaneously by the web service host machine. In other words, at 1000 TPS exactly 1000 virtual users (20 per client machine) with a one-second think time are driving load against the server.
(1 second think time, many simulated users--varies for each product to achieve peak throughput for each)
(No think time, 16 threads)
Note GetOrder is not included using this single-client methodology as it results in a client-side Mercury LoadRunner bottleneck, preventing full saturation [100% CPU utilization] of any tested product. This client-side bottleneck is not present when using 50 distributed clients to drive load. Interestingly, only the GetOrder method is affected by a client side bottleneck using the single-client test.
The results indicate that Web Service performance is roughly 25% better in .NET 2.0 than .NET 1.1 when the soap object size is large and hence deserialization/serialization operations are more intensive. The difference is even more dramatic for smaller SOAP message sizes, with the echostruct size 20 test showing .NET 2.0 beta2 performance to be roughly 40% better than .NET 1.1. In all cases using the 50-client methodology, .NET outperforms both SunOne and IBM Websphere 6.0, often by wide margins.
The Web containers chosen for these tests and used with Sun’s JWSDP 1.5 and IBM Websphere 6.0 are the http servers recommended by each vendor for enterprise deployments (SunOne HTTP Server and IBM HTTP Server respectively). With the code published, customers can configure and run the tests using different web servers and/or application servers. For example, TomCat can be used with JWSDP very easily, and as well IBM’s in-process HTTP listener (port 9080) can be used in lieu of the IBM HTTP Server. It should be noted however, that neither of these products are the products officially recommended by the vendor for actual enterprise deployments, in part because of lacking management features and in part due to the in-process nature of the java code with respect to the actual web server. In the case of .NET, the default process model is used, meaning in all cases there is full process-isolation between the .NET web service logic and the actual web server itself. This is important for reliability/crash protection. In all cases, a single JVM instance/.NET Worker process is sufficient to achieve full server saturation across both test methodologies and all tests.
Web Service performance has been significantly improved in .NET 2.0 vs. .NET 1.1. The WSTest 1.1 Benchmark kit, downloadable from MSDN, is a good tool to use to measure web service performance of different application server platforms, and to judge raw capacity of backend hardware for processing web service requests. Customers are encouraged to download the kit (inclusive of Mercury Scripts) and perform their own tests.
IIS 6.0
IIS access logging turned off
IIS Authentication set to anonymous
.NET via Web.Config
Authentication set to none to match java authentication
Session state turned off
JWSDP 1.5
In Server.XML the following options are used
-xmx1024
-xms1024
"-server"
SunOne HTTP Server 6.1
Session state turned off
Access logging turned off
The following options are used in Magnus.config:
RqThrottle 128
KeepAliveThreads 2
KeepAliveQueryMeanTime 5000
KeepAliveQueryMaxSleepTime 5000
Security Off
ListenQ 2000
Access Logging commented out
KeepAlive 3000
WebSphere
Java Process Definition: -xmx1024 and -xms1024
No access logging
No Session state
Inbound Http channel using 3000 keepalives
Web container set to 50 min and max threads
IBM HTTP Server (via httpd.conf)
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 10
<IfModule mpm_winnt.c>
ThreadLimit 2048
ThreadsPerChild 250
MaxRequestsPerChild 0
</IfModule>
1. EchoSynthetic is not included in the Microsoft WSTest 1.1 benchmark kit to keep things in xml/text form (vs. binary data) in the LoadRunner client scripts. Generating the appropriate soap binary requests from LoadRunner is likely possible, but outside the scope of this testing. Should customers want to run EchoSynthetic, they can easily do so by downloading the WSTest 1.0 benchmark code from Sun’s site and using their custom java driver program.
WS-TEST1.1.zip
Comparing XML Performance
. NET 2.0 Beta2, . NET 1.1,
and Sun Java 1.5 Platforms
The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.
This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, AS TO THE INFORMATION IN THIS DOCUMENT.
Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.
Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.
Ó 2001 Microsoft Corporation. All rights reserved.
Microsoft, the .NET logo, Visual Studio, Win32, Windows, and Windows Server 2003 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries.
The names of actual companies and products mentioned herein may be the trademarks of their respective owners.
Microsoft Corporation • One Microsoft Way • Redmond, WA 98052-6399 • USA
1297
XML Mark is an XML parsing benchmark originally created by Sun Microsystems to demonstrate the relative performance of Java XML parsing vs. . NET (C#) XML parsing. The original Sun XML Mark 1.0 benchmark kit can be downloaded from Sun’s Website at: http://java.sun.com/developer/codesamples/webservices.html#Performance.
Sun’s original accompanying whitepaper can be downloaded at: http://java.sun.com/performance/reference/whitepapers/XML_Test-1_0.pdf
With the release of . NET 2.0 Beta2 on MSDN, we used Sun’s XML Mark benchmark to compare the XML parsing performance of:
- Sun J2SE (JVM 1.5) with Sun’s latest Java Web Services Developer Pack (JWSDP) version 1.5
- . NET 1.1
- . NET 2.0 Beta 2 release (publicly available on MSDN)
Sun concluded in their original tests (using JVM 1.4/JWSDP1.3 vs. . NET 1.1) that the Java platform significantly outperformed . NET in both the pull/streaming tests (SAX tests), and the XML build tests ( DOM tests) included in the XML Mark kit. While Sun’s JVM 1.5 does outperform .NET 1.1 in the benchmark tests, the difference is exaggerated based on two significant issues that are now fixed in our XML Mark 1.1 implementation:
- The java code included in their kit has a bug that results in no serialization (save to disk) occurring in the java tests even when the run properties are set to serialize the XML test document, while the .NET implementation properly saves to disk when the serialization parameter is set.1
- We also found that Sun is using the GetElementsByTagName method in C#, and this results in a mismatch of the C# and Java versions. Although both harnesses use XmlElement.GetElemensByTagName(), the C# implementation of this method keeps track of a live list of nodes which is negatively impacted by some of the DOM test scenarios that have edits. A better method for element selection in the C# harness is XmlElement.SelectNodes() which matches the Java harness in functionality.
XML Mark 1.1 fixes both of these issues in the benchmark kit which otherwise represents a good test harness for both Java and C# for XML parse performance. In this paper, we publish the results of testing . NET 1.1, . NET 2.0 Beta2 and Sun’s J2SE 1.5 platform with this new XML Mark 1.1 benchmark kit. The results show some very significant performance improvements in XML parsing between . NET 1.1 and . NET 2.0.
As with other . NET benchmarks, we are providing full source code, build commands and install/run instructions for both the Java and C# implementations, and it’s an easy test for any customer to quickly download, install and run on the hardware of their choosing.
The following description of XML Mark is taken directly from Sun’s original whitepaper2:
XML Test is an XML processing test developed at Sun Microsystems. It is designed to mimic the processing that takes place in the lifecycle of an XML document. Typically that involves the following steps:
• Parse - Scan through the XML document processing elements and attributes and possibly build an in-memory tree (for DOM parsers). Parsing is a pre-requisite for any processing of an XML document.
• Access - Extract the data from the elements and attributes of parts of the document into the application program. For example, given an XML document for an invoice, the application might want to retrieve the prices for each item in the invoice.
• Modify - Change the textual content of elements or attributes and possibly also the structure of the document by inserting or deleting elements. This does not apply to streaming parsers. As an example, an application might wish to update the prices of some of the items in an invoice or may want to insert or delete some items.
• Serialize - Convert the in-memory tree representation to a textual form that is written to a disk file or forwarded to a network stream. This makes sense only for tree building parsers and is necessary in the cases where the XML document has been modified in memory.
XML Test simulates a multi-threaded server program that processes multiple XML documents in parallel. This is very similar to an application server that deploys web services and concurrently processes a number of XML documents that arrive in client requests. Since we wanted to concentrate on XML processing performance, rather than use some sort of web container, we designed a standalone multi-threaded program implemented in both Java and C# that processed XML document files. To avoid the effect of file I/O, the documents are read from and written to memory streams. XML Test measures the throughput of a system processing XML documents. The notion of an XML transaction here corresponds to a complete lifecycle of an XML document. For tree building parsers this requires the four steps of parse, access, modify and serialize
while for streaming parsers it just involves parse and access. XML Test reports one metric: Throughput - Average number of XML transactions executed per second.
The XML documents used with XML Test are based on a business invoice document. Each invoice has a fixed length header and summary and a variable number of lineitems. The schematic of the invoice schema is given below in XML schema form.
<complexType name="InvoiceType">
<complexContent>
<sequence>
<element name="Header" type="InvoiceHeaderType"/>
<element name="LineItem" type="InvoiceLineItemType" minOccurs="1"
maxOccurs="unbounded"/>
<element name="Summary" type="InvoiceSummaryType"/>
</sequence>
</complexContent>
</complexType>
Though this schema can be used to generate an invoice document of almost any size, we made use of two particular document sizes in our experiments. Streaming parsers are typically used to process large XML documents, and so to compare SAX and the pull parser, we used an invoice with 1000 lineitems (about 900 KB). Tree-building DOM parsers have to operate under memory constraints since they construct a representation of the whole document in memory. Therefore for the DOM parsers we used a smaller invoice document containing 100 lineitems (about 90 KB).
Based on the above schema the lifecycle of an XML document as applicable to XMLTest can be defined as follows.
• Parse - Build a complete document tree in memory (in the case of DOM). In the case of streaming parsers, scan through the XML document.
• Access - Retrieve the Currency attribute and the PriceAmount for the number of LineItems specified.
• Content Modification - Increase the PriceAmount for all lineitems by 10%. Not implemented for SAX / pull parser.
• Structure Modification - Delete a specified number of lineitems and insert the same number at the end of the set of LineItems. The new lineitems must have LineIDs in the properly increasing sequence. Not implemented for SAX / pull parser.
• Serialize - Convert the entire in-memory tree to a serialized XML form written to a stream. Not implemented for SAX / pull parser.
Test Details
XML Test has been used to characterize these XML parsers:
Java . NET
Streaming SAX Pull
Tree Building DOM DOM
XML Test can be configured using the following parameters:
- Number of threads - This is tuned to maximize CPU utilization and system throughput.
- StreamUsage – Whether stream parsers are being tested.
- DOMUsage – Whether DOM parsers are being tested.
- Selection - The percentage of lineitems retrieved in the access phase.
- ContentModification - The percentage of lineitems whose textual content is modified.
- Structure Modification - The percentage of lineitems inserted or deleted from the DOM tree.
- RampUp - Time allotted for warmup of the system.
- SteadyState - The interval when transaction throughput is measured.
- RampDown - Time allotted for rampdown, completing transactions in flight.
- XmlFiles - The actual XML documents used by XML Test.
XML Test reads these properties at initialization into an in-memory structure that is then accessed by each thread to initiate a transaction as per the defined mix. To keep things as simple as possible, XML Test is a single-tier system where the test driver that instantiates an XML transaction is part of each worker thread. A new transaction is started as soon as a prior transaction is completed (there is no think time). The number of transactions executed and the response time is accumulated during the SteadyState period and is reported at the end of the run.”
In the test results published below, we used a Hewlett Packard HP DL585 AMD Opteron machine running 32-bit Windows Server 2003. The machine is configured as follows:
- 2 x 1.8 GHZ AMD Opteron Processors
- 3 GB of RAM
- 36 GB SCSI Drive array (10 ms access time)
- Windows Server 2003 Standard Edition
- . NET v1.1.4322 (included with Windows Server 2003)
- . NET v2.0.50215 (public Beta 2 which can be downloaded from MSDN for free)
- Sun J2SE 1.5 Update 2 (JVM version 5.0.20.9)/Sun JWSDP 1.5
DOM Build Test Results with No XML Serialization
In these tests, Sun’s sample .INI files for DOM1, DOM2 and DOM3 were run unmodified as follows:
DOM1.INI:
[CONFIG]
Agents = 4
PullParserUsage = 0
SAXUsage = 0
DOMUsage = 100
Selection = 25
Validation = 0
ContentModification = 0
StructureModification = 0
Serialization = 0
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv100.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
DOM2.INI:
[CONFIG]
Agents = 4
PullParserUsage = 0
SAXUsage = 0
DOMUsage = 100
Selection = 50
Validation = 0
ContentModification = 0
StructureModification = 0
Serialization = 0
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv100.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
DOM3.INI:
[CONFIG]
Agents = 4
PullParserUsage = 0
SAXUsage = 0
DOMUsage = 100
Selection = 100
Validation = 0
ContentModification = 0
StructureModification = 0
Serialization = 0
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv100.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
In these tests, Sun’s sample .INI files for DOM4, DOM5 and DOM6 were run unmodified as follows:
DOM4.INI
[CONFIG]
Agents = 4
PullParserUsage = 0
SAXUsage = 0
DOMUsage = 100
Selection = 50
Validation = 0
ContentModification = 25
StructureModification = 0
Serialization = 100
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv100.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
DOM5.INI
[CONFIG]
Agents = 4
PullParserUsage = 0
SAXUsage = 0
DOMUsage = 100
Selection = 50
Validation = 0
ContentModification = 0
StructureModification = 25
Serialization = 100
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv100.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
DOM6.INI
[CONFIG]
Agents = 4
PullParserUsage = 0
SAXUsage = 0
DOMUsage = 100
Selection = 50
Validation = 0
ContentModification = 25
StructureModification = 25
Serialization = 100
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv100.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
In these tests, Sun’s sample .INI files for SAX1, SAX2 and SAX3 were run unmodified as follows:
SAX1.INI
[CONFIG]
Agents = 4
PullParserUsage = 100
SAXUsage = 100
DOMUsage = 0
Selection = 25
Validation = 0
ContentModification = 0
StructureModification = 0
Serialization = 0
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv1000.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
SAX2.INI
[CONFIG]
Agents = 4
PullParserUsage = 100
SAXUsage = 100
DOMUsage = 0
Selection = 50
Validation = 0
ContentModification = 0
StructureModification = 0
Serialization = 0
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv1000.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
SAX3.INI
[CONFIG]
Agents = 4
PullParserUsage = 100
SAXUsage = 100
DOMUsage = 0
Selection = 100
Validation = 0
ContentModification = 0
StructureModification = 0
Serialization = 0
#Rampup time for the run in seconds
RampUp = 180
#SteadyState time for the run in seconds
SteadyState = 300
#Rampdown time for the run in seconds
RampDown = 30
#Total number of XML documents for processing
XMLDocCount = 1
#Actual xml file names with complete path
XML_ FILE_1 = ..\\..\\docs\\inv1000.xml
SchemaFile = ..\\..\\docs\\sm-inv.xsd
OutputDir = ..\\..\\results
OutputFilePrefix = XmlMarkSummary
The XML Mark 1.1 Benchmark Kit demonstrates that major improvements have been made with . NET 2.0 over the . NET 1.1 runtime for XML parsing. These improvements will impact both client and server programs that do any amount of direct XML manipulation. Furthermore, the results indicate that . NET 2.0 performs significantly better than the latest Sun Java 1.5 platform in XML parsing scenarios. We encourage customers to download the XML Mark 1.1 benchmark kit and test the scenarios for themselves.
1. Sun’s code accidentally used the validation % property instead of the serialization % property to determine what serialization parameters to run with. The C# version in their kit uses the proper initialization property as specified by the user. The result is that in using their 1.0 kit the Java and . NET results cannot be compared for any test where the serialization % is set different to the validation % as specified by the user in the initialization (.ini) files. Specifically, when running Sun’s version as downloaded from their site, their DOM4, DOM5 and DOM6 tests are invalid since . NET is performing serialization for every processed document and Java is performing no serialization at all. The corrected results for Java 1.5 are shown in this paper based on our fix included in XML Mark 1.1.
2. See XML Processing Performance in Java and . NET, published by Sun Microsystems in June 2004 and downloadable at: http://java.sun.com/performance/reference/whitepapers/XML_Test-1_0.pdf
XMLMark1.1.zip
Authors
 | (C) 2004 Microsoft Corporation. All rights reserved. Microsoft, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners. |
|
