The search for the perfect Web Service testing tool

Here at Corillian I am responsible for testing of the product Intelligent Authentication (IA). IA is a .NET Web Service written in C# and it is the first
Corillian product released that is a Web Service. IA is also the first Web Service that I've ever tested. When I accepted the QA position on the Corporate Security team I knew I had some new, unique challenges ahead me with finding testing tools and techniques to successfully test a Web Service. We've released 1.0 and are now working on "1.1". I spent a good portion of the 1.0 development time trying to find the perfect application and thus, here is my story:

First off, what was I looking for? My test tool needs weren't too complex:

1. Create and combine test cases to run sequentially
2. Provide a test case status and overview
3. Allow data driven testing (static and dynamic)
4. Compare regression history
5. Allow the ability to send SOAP calls that aren't compliant with the WSDL
6. Ability to validate valid SOAP responses and SOAP faults

I started by asking around which revealed little, and then perusing the Web. I found and looked through half a dozen home grown tools but nothing went beyond consuming a WSDL and sending requests to it. The tools I've listed below are worthy of a mention and the last tool listed is highly recommended (ParaSoft SOATest):

MindReef SOAPScope (achieves goals 5 & 6)
Everybody on the IA team has SOAPScope and recommended that I start there. That I did. Playing around with the tool revealed that it is great for creating SOAP traffic but it didn't take long to realize that this tool was designed for developers and not testers who wanted to store test cases. Since SOAPScope runs in Internet Explorer I toyed with the thought of driving the interface with Segue SilkTest and using SilkTest to store my test cases. This seemed like a lot of work for testing something as simple as a Web Service. There has to be a better solution for testing out there...

Segue .NET Explorer (achieves goals 1, 2, 1/2 of 3, 6)
Corillian has used Segue products for quite a few years now; SilkTest and Silk Performer. A recent release of Silk Performer also provided an application with it called .NET Explorer. This app provides a quick and easy way to create Web Service test scripts without any .NET programming knowledge. I was able to quickly point to the WSDL, see all the methods, and then start creating test cases within the GUI. First impression: Wow! This is easy AND I can store & track my test cases! Second and last impression: Wow! This app is full of defects (I submitted 6 to Segue, 2 critical/showstoppers). After about 2 weeks I felt like I was testing 2 apps; .NET Explorer and Intelligent Authentication. It had a lot of little quirks that I found myself working around every day. The final straw for me was when the WSDL revealed a major change and the app didn't know how to react; the result was that I lost all of my test cases/scripts that I had built over the last two weeks. Ouch... Fortunately I had them documented so that I could rebuild in whatever new test app I could find. NEXT!

MindReef SOAPScope and Sputnik Beta (2, 4, 5, don't remember if it did 6)
Now I was worried. I was already half way through the project and I didn't have a reliable tool. I felt good about my testing progress but my automated regression suite was gone. Not good when I'm the only tester. I quickly went back into searching mode, looking high and low on the Web for any tool that dealt with SOAP. Around the same time, my manager Greg Hughes pinged Scott Hanselman to see if he had seen or heard about tools for SOAP. Scott had just received an email about Beta testing a product name Sputnik. Ahh, a ray of light from the stormy clouds looming over my head. I installed the Beta and found that Sputnik wasn't really much different than SOAPScope. It used SOAPScope but it had a bit more to it. The addition allowed me to save a SOAP request and response and treat it as a test case that could be run again. This was a step forward from SOAPScope but it didn't offer much more than that. The application was slow and clunky and didn't give me much hope. It just wasn't enough, I actually felt better about using .NET Explorer. After already losing once I wasn't going to bet the bank on a beta product. Time to move on. Since then, Sputnik has become "MindReef Coral" and I haven't had a chance to review whether or not it gained more features since the beta.

ParaSoft SOATest (1, 2, 3, 4, 5, 6)
More and deeper searches on the Web, with creative key words, eventually revealed a product by Parasoft called SOAPTest (recently renamed SOATest). Their Website revealed to me that it did everything I needed it to do. It seemed too good to be true after all the research and trials that I did but there was only way to find out. I installed SOATest and waited for the call from the representative who had my trial key. My expected call ended up being a game of phone tag (I hate getting keys this way) and I ended up with the key a couple days later. Once I had my key I spent a few hours over the weekend putting the app through the test; making sure it could do everything all the other apps could do, and attempting to do all the things I needed the tool to do. The learning curve was quite a bit steeper than the other apps but for a good reason, this application is by far more robust. It does everything I need and more. There are too many things to spell out myself so here is the blurb from Parasoft's site:

• Ensure the reliability, quality, security and interoperability of your Web service.
• Penetration testing integrated with functional testing for complete coverage.
• Uniform test suites can be rolled over from unit testing to functional testing to load testing to security testing.
• Prevent errors, pinpoint weaknesses, and stress test long before deployment.
• Verify data integrity and server/client functionality.
• Identify server capabilities under stress and load.
• Accelerate time to market.

• Scriptless Web Services testing.
• WS-Security, SAML, Username Token, X.509, XML Encryption, and XML Signature support.
• UDDI support: query verification, validation, and load testing.
• Automatic test creation from WSDL, WSIL, UDDI and HTTP Traffic.
• Asynchronous Testing: JMS, Parlay (X), SCP, WS-Addressing support.
• Complete coverage testing workflow through complex scenarios and multiple service endpoints.
• WSDL schema and semantic verification and compliance to WS-I Basic Profile 1.1.
• Data-driven testing through data sources (Excel, CSV, Database Queries, etc).
• MIME Attachment support.
• Windows Perfmon, SNMP, and JMX Monitors.
• Detailed Report generation in HTML, XML and Text formats.
• Real-Time graphs and charts.

• HTTP 1.0
• HTTP 1.1 w/Keep-Alive Connection
• HTTPS
• TCP/IP
• JMS

• Windows 2000/XP
• Linux
• Solaris

See what I'm saying? A ton of stuff! So here I am months and 1000+ automated test cases later. I love it. It works. It's easy. It's robust. It makes build regression a snap for me. SOATest is the bomb! It only has a few small downfalls:

• The price is a little large for small companies (around $4000 a seat).
• Scripting can be done with JavaScript, Java, and Python. That's not so bad, but the scripting capability is wrapped by the app so you are limited on some things (for example you can only pass two parameters in a JavaScript function). Use of JavaScript can be a bit painful if you're used to programming for IE and Netscape. The type and version of JavaScript they include with the app will leave you scratching your head when trying some simple things.
• Occasionally the window frames don't refresh correctly (it's a Java application).

There aren't a ton of robust Web Service testing tools out there, so hopefully this post saves you some time in your search and decision!

Update 1/16/2006: SOATest wins "Best Testing Tool" in the "Oscars of the Software Industry" (Sys-Con)