An automation engineer's test automation progress is often a black box to the project team and managers and that is serious "egg on the face" for any automation initiative. One day while automating I started reminiscing about how I used to monitor and report test case status while doing functional testing, and thought to myself "How can I do that with my test automation?". Shortly after. a process and a tool was born, and stats were included in my weekly reports. I also had the ability to provide detailed test descriptions. Now others had insight to my goal, my progress, I could estimate a completion date, and the project team could review my test descriptions looking for voids in coverage. A bonus benefit to tracking status is that multiple automation engineers can work on one project and not accidentally stomp each other. Seems like a no-brainer right?.. But more often than not I see automation engineers working in an automation black box leaving them unaccountable to all.

Here is an example of how I make myself and my test automation accountable:

1. I stub out my test cases when reviewing requirements (the final number is my goal). For example, each test case is usually one method in my automation test suite. One hundred tests equates to 100 methods. I use separate classes to segregate functionality. My method name follow a pattern and are very descriptive, which helps me decipher what they are when they are in large lists and allows for easy alphabetical sorting.
2. When stubbing the tests/method, I write the test description/steps with it's verification points. For example, in the screenshot below, the "Description" attribute contains these details.
3. I track test/method development status. In the example below you can see the various status that I use. Status is the key to monitoring progress!
4. I tie defect ids or agile task numbers to test cases, which makes for easy searching when I'm doing defect regression:
    
   
5. Finally, I use a tool/automation to extract goal, status, and test description: Note that in the above "Stats" screenshot I have a Test Summary "Count" which is my goal, I have a count of the various states, and have a percentage of the various states. "Completed" percentage is my progress towards the goal. I typically take a screenshot of this tab and paste it into my status report.
   
   Note that in the above "Test Details" screenshot, I have a column for Class and Method which allow me to sort by them. Then I have a test "Description", the test "State",  the "Reason" for test blockage, and finally a place for "Comments". This tab is nice for a quick overview of tests, it allows sorting which is nice if you want to, for example, sort by "Blocked". This can also be exported into an Excel spreadsheet. This view is VERY helpful when you end up having hundreds of automated tests, because scrolling through hundreds of lines of code can make things easy to miss or can get confusing.

 
The 5 points made above were done in my .NET test automation environment which uses a custom attribute I created called "TestProgress". The reporting GUI uses reflection to extract the class, method, and attribute details. The example is for .NET but this process and pattern could be used in any language that you may be automating in. For example in a scripting language (e.g. Ruby), you could provide "Test Progress" as a comment  above the method and then use regular expressions to parse the files to create your report. For example, the Test Progress comment could look something like:

2009. What an eventful year. Eventful in my personal life as well as in my SQA career. A good, eventful year.

I didn't blog much in 2009, 17 posts in all, and no topics that were SQA groundbreaking.  Yeah, I'm pretty much ashamed of myself and have watched my blog fall off peoples' radar.If I were to highlight my favorite post it would be my turn from SQA words to illustrations with Do Loop Until Zero. A hit or a miss, I don't know; I don't get comments either way on this blog. But none the less, it's something I enjoy doing. Hopefully you guys will see more of this "comic", if all works out well, it will be in the 1st issue of the new and upcoming Software Testing Club magazine.

Though the blog was quiet, my SQA and testing career wasn't. In the last year I had the ability to start filling a large gap that was present in my testing experience portfolio. Prior to 2009 I had no experience in the Linux world and the technologies that surrounded it. Joining a new group within GoDaddy quickly changed this. In 2009 I did a 180 degree turn from my beloved Windows world and submerged myself in Linux in an effort to test and automate a new, internal product. I was scared to make the jump, mostly because my Windows wisdom would be put to little use, and my lack of Linux knowledge would make me a slower tester and automator. Not so enticing when I really pride myself on speed and efficiency ("Hire me, Hire ME! I'm two testers for the price of one!"). Scared or not it was an awesome opportunity to further my skills, and help a 1.0 product using my experience with agile practices and automation. With the help of an awesome two  man development team, I was able to learn, automate and wade through the following technology highlights in 2009:

Product: A storage system (C, mySQL):

• I used PuTTY as an SSH client to the dev, test and prod environment running CentOS as a flavor of Linux
• I extended developer unit tests and automated API functional and boundary testing with Perl unit testing (Test::Unit)
• I extended PHPUnit to serve as an automation framework for automation of functional tests (use case, boundary, error handling,etc). The framework was named TAEL (Test Automation Ecosystem for Linux).

Product: FTP Server that uses the storage system (Perl, mySQL)

• I automated use cases, and FTP functions using TAEL. FTP functionality was tested using PHP's FTP  library. Validation was done through FTP responses, and mySQL queries.
• I performance tested the FTP server and underlying storage system with Apache JMeter. FTP in JMeter is not very robust, and worse yet forces a connection open, logon and close for every request needed, which is not very realistic. Thankfully it's open source (Java) so I extended it and tweaked it to fit our needs.

Product: User Management Web Service

• I automated use cases, boundaries, etc with TAEL. Validation was done by querying mySQL or parsing the Web Service response using XPATH queries.

Tool: User Experience Monitor

• In an effort to monitor response times on an ongoing basis, I wrote a script that executes basic functionality every 15 minutes, stores the timed results in FTP, where they are picked up and  processed by a chron job that puts the results in a database. Chron takes the results puts them into an XML format which are then viewed in a PHP web page using the chart control XML/SWF charts. We found some very interesting activity and trends through this test/monitor. This turned out to be a very interesting almost real-time QA asset for the team.

Product: REST service

• Automated REST service tests using the Visual Studio 2008 unit testing framework (MSTest, Microsoft.VisualStudio.TestTools.UnitTesting namespace)
• Created a .NET API for the REST service that would serve as an easy to use interface for automated testing as well as others wanting to use the service from .NET.
• Performance tested the REST service using my .NET API and Visual Studio 2008 load testing framework
   

Automation with Ruby: With a department wide goal that everybody must do a little automation, I led them down the path of Ruby/Watir (due to cost, and Ruby being pretty easy to learn). The results are looking pretty good, adoption has gone well and progress is being made. Here are a few details about the framework that I built over a few weekends:

• Uses a pattern that I call "Test, Navigate, Execute, Assert"
• Manages tests with the Ruby library: Test::Unit
• Uses Watir for web page automation
• Web Service automation is done with:  soap4r  & REXML
• MySQL database validation with the gem:  dbd-mysql
• Data driven automation from Excel using Roo

 
Process: Since I've been lucky enough to work with a highly motivated, small team of three, our process needs to be and has been really light. We've been pretty successful at being extremely agile. For project management we followed a scrum-like process for a little over half a year using the tool Target Process, but then moved to a KanBan approach with our own home-grown tool. Recently we moved from the home-grown tool to a trial with Jira, while trying to maintain the project in KanBan style. I have to  say that I really like KanBan, it works particularly well for our team because it is small. When you're as small and tight knit as out team is, we always know what each other is working on, so the more light-weight the better. It seems the largest payoff of these types of process and tools for our team is tracking backlog items as well as giving management insight to what we're up to.

What's in store for me in 2010? Well, I'll likely be working on the same products, but as far as major learning and growth opportunity I'm excited to dive into the awesome new features of Visual Studio 2010 for Testers as well as to learn and use some C++. Now, if I can just convince myself to blog about those experiences as I go.

With help from Gerhard, a QAInsight reader, the User Agent Switcher MONTSTER XML file has been updated to use the new, folder feature. Also, another IPhone user agent has been added as well as one for Chrome. As always, the permalink is here, and can be found in the right navigation under the "My Testing Tools" header, link: "User-Agent Info and Tools".

MSDN documentation for "Testing the Application" in VSTS 2010:
http://msdn.microsoft.com/en-us/library/ms182409(VS.100).aspx

Video: Functional UI Testing with VSTS 2010
http://channel9.msdn.com/shows/10-4/10-4-Episode-18-Functional-UI-Testing/

How to add a VSTS 2010 coded UI test to a build:
http://blogs.msdn.com/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx

Creating and running a VSTS 2010 coded UI test through a Lab Manager project:
http://blogs.msdn.com/jasonz/archive/2009/05/26/vs2010-tutorial-testing-tutorial-step-2.aspx
http://blogs.msdn.com/mathew_aniyan/archive/2009/05/26/coded-ui-test-from-microsoft-test-lab-manager.aspx

Explanation of the various Test tool names and products:
http://blogs.msdn.com/jasonz/archive/2009/05/12/announcing-microsoft-test-and-lab-manager.aspx

VSTS related blogs:
http://blogs.msdn.com/vstsqualitytools/
http://blogs.msdn.com/amit_chatterjee/ 
http://blogs.msdn.com/mathew_aniyan/

Thanks to co-worker Julio Verano for passing this on:

Here is a 17 minute video from MIX09 on Automated User Interface Testing with Microsoft Visual Studio Team System 2010.

VSTS 2010 looks like it has great potential for testers and developers, but I think Microsoft is still behind in browser automation and functionality when comparing to the great work from Art of Test with WebAii and Design Canvas.  I'm excited that Microsoft is actively pursuing and growing in this space though. It is needed badly! Here is the current proposed platform support for VSTS 2010:

A question from Tobbe Ryber has inspired me to jot down a few things I've been meaning to do in a more extensive blog post for a long time. But since it hasn't happened yet, I figure it probably never will, so you'll have to settle for my abbreviated, half-ass version.

Twenty reasons to use Visual Studio Team System 2008 Test Edition for your software testing automation framework, ESPECIALLY if your development team is using .NET and Visual Studio:

1. You are using the .NET platform which is a set of stable and robust libraries that allow you to do just about anything. Make HTTP requests, make Web Service requests, use COM, make database queries, the list goes on and on. Basically anything your .NET developer is doing you're going to be able to tap into using the same context. Easily.
2. You have the ability to easily make calls into several layers of the application under test using a few lines or code, without duct taping and bailing wiring a bunch of libraries or technologies together. Imagine automating something in the browser, making a call to a Web service and then calling the database to validate your results...All in a few lines of code.
3. There are awesome tools and libraries that are built on .NET that allow you to automate browsers, such as SWEA, WatiN, and HTTPWatch.
4. There is a great library and Visual Studio add-on that allows you to automate multiple browsers (IE 7, 8, FireFox. Safari and Chrome any day now), as well as Silverlight. Best yet, the recorder integrates with the IDE: ArtofTest's WebAii and Design Canvas.
5. Your 'test harness' is built into the IDE, and your tests can also be ran from the command line.
6. The IDE is top notch when it comes to development and debugging (and test development and debugging). I've been using VS for automation since VS 2005, and when I've had to automate in other worlds (e.g. Linux, PHP, and Perl) I honestly feel like I'm working with tools that equate to a chisel and stone tablet.
7. Auto-complete in the IDE is a huge timesaver. Your time spent searching the internet or referring to a library's specifications is far less with auto-complete.
8. Syntax issues with scripting languages (JavaScript, Ruby, etc.) can be a huge waste of time at runtime. If you write a test that runs for minutes, hours, or days it could fail halfway through due to syntax. A compiled language is not going to do this.
9. The Microsoft.VisualStudio.TestTools.UnitTesting namespace is not just for unit testing, it works great for test automation. It feels a lot like nUnit to me.
10. Integrating your tests with development builds is cakewalk. Using the mstest command line, it's easy to have your tests run with a build in TFS or CruiseControl.
11. You have the ability to easily move some of your tests up the chain to run along side of developers' unit tests. By doing this you now have automated acceptance tests, so that releases to QA have higher quality.
12. You are using the same environment/language as your developers. By doing this, you gain:
13. A. The ability to have developers help you with getting over the .NET language or VS IDE learning curve.
14. B. Knowledge and use of the same language and libraries used for development, thus having a greater technical understanding of what you're testing.
15. C. Easily share and discuss your tests with developers because they are familiar with the language you are using.
16. Test results are in an XML format which means that if you want to use something other than VSTS to view results you can easily manage it.
17. The .NET community is huge. Help, technical examples, and issue-workarounds are an Internet search away.
18. Examples are SUPER helpful through MSDN. Training video series such as How Do I and VSTS Learn are a great alternative.
19. VSTS also does load testing.
20. .NET, C#, VB.NET, and Visual Studio experience on your resume are technology skills and buzzwords that lure recruiters.

Web-site layout, one of the many things that can keep a tester busy. Uhm...overwhelmed? So many browsers so little time... Wouldn't it be nice if you, the mighty tester, could just review a butt-load of screen-shots of your Web application in multiple browsers on multiple platforms to make sure there are no layout issues?  Wouldn't that be quick and efficient?

I've got 2 semi-solutions for you (keep in mind I've done little with both, so forgive an misinformation):

Litmus
"We've felt the pain of getting website designs to work correctly across different browsers. Not to mention designing email newsletters that work on all email clients. Litmus makes compatibility testing easier. Litmus is lightning-fast, reliable and affordable. It's relied upon by thousands of smart freelancers and switched-on agencies; as well as big companies like Yahoo!, Facebook and eBay."

The FREE part of Litmus: Screen-shots of your site in IE 7 and FireFox 2.

The $ part of Litmus: Pay $24 a month to get 23 browsers and 14 email clients.

BrowserShots
"Browsershots makes screenshots of your web design in different browsers. It is a free open-source online service created by Johann C. Rocholl. When you submit your web address, it will be added to the job queue. A number of distributed computers will open your website in their browser. Then they will make screenshots and upload them to the central server here."

The FREE part of BrowserShots: 70 browsers on various platforms! Submissions get dumped to a queue for processing.

The $ part of BrowserShots: Pay $15 a month to get priority processing.

Both of these appear to be good services that can provide quick insight to layout problems in your site. However, as far as I can tell the two big limitations are:

• You are limited to pages that you can navigate to via URL, which rips the grandiose dream of having a screen-shot for every page in your website (pages that require form post or special conditions to get to are not going to happen). However, Litmus does provide a step in the right direction with it's functionality for authentication.
• Your site must be exposed to the Web, doing little for internal Dev and QA project cycles.

I have a dream...

• I want to screen-shot any page in my website (requires a decent engine that will allow me to get to the various pages in my site).
• I want to screen-shot my site that is not yet published for the world (requires the service to exist within my local network).
• Once I've approved an ideal layout screen-shot I want the software to determine and tell me if the other screenshots are worth looking at (by doing a statistical image comparison with a predefined pass/fail threshold).
• I want to provide basic wire-frame definitions and have software determine if my screen-shots are within reason (by analyzing elements in the DOM and browser dimensions)
• Get rid of screen-shots and do DOM to DOM element width and height comparisons between browsers (Come on, it's a dream, standards compliance for all browsers (another dream) might make it possible?)

Honestly, I think the dream is doable... So many dreams/ideas, so little time.

For those of you in need of doing SQL load testing from Visual Studio 2005 or 2008 there is a new open source project at CodePlex called SQL Load Test. How does SQL Load Test work?

"This tool takes a SQL Profiler trace file and generates a unit test that replays the same sequence of database calls found in the trace file. The unit test is designed to be used in a Visual Studio Load Test. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing."

Get more info and download SQL Load Test here.

Over the last month I've been looking at Zephyr, a test management system that touts itself as "Next Generation". What exactly is Zephyr and what does it have to offer to the testing community?

"Taking a completely realistic approach to how Test Teams work, collaborate and interact with each other in their department and the rest of their world, Zephyr brings together a comprehensive set of features, a really slick UI and Web 2.0 features at a price point that makes it very affordable for all team sizes.

Zephyr is based around the concept of Desktops & Dashboards. Every role in a Test Department has a customized Testing Desktop with relevant applications that allow them to do their jobs faster and better, as they all share data from a centralized repository and communicate via a collaborative backbone. Dashboards are automated and live, keeping the whole company updated on every aspect of testing and product quality."

At a high level Zephyr offers:

• Testing Desktop
• Dashboards
• Metrics & Reporting
• Test Case Repository
• Resource Management
• Project Management
• Release Management
• Test Case Creation
• Test Execution Planning
• Test Execution
• Document Management
• Defect Tracking
• Collaboration
• Import and Export
• User Interface
• Integration

The 20,00 foot view of Zephyr is this (see the related "How Zephyr works" video here):

Now, I've been managing test cases in a Excel spreadsheet for years, a fairly advanced one at that. It utilizes Visual Basic for Applications (VBA) and gets the job done very well for me, several other QA Engineers, and a few managers. So, having to consider a test management system that costs $$ it a hard to swallow...If it's not broke, don't fix it right? With that eating at me, when looking at Zephyr I decided to compare it to what I have and currently use. Let me tell you what my KISS test management system consists of:

• Quick and easy test case writing
• Consistent test plan and case format
• Reusable test case library that contains commonly used test cases
• Brief but technical test case writing format
• Test case state statistics by section and total
• Testing summary for all testing sections/worksheets
• Test case trends via charts
• Automated coloring of test case status for quick visual reference
• Automated test case to build mapping
• Test case to defect mapping
• Simultaneously sharing between multiple testers
• Tester assignment by section/worksheet

Granted, it's not perfect, but again it works very well and people really like it. With my testing world as the level set, let's jump into the good, the bad, and the things to think about if your considering Zephyr:

Tester Assignment
You can specify users to run specific test cases or whole sections. Very nice!

Copying
I can drag and drop individual cases from one folder to another, but I can't figure out for the life of me how to drag sub-folders of test cases into another folder (they move not copy). I also was unable to successfully import a previous export. If indeed this is possible it's not easy or intuitive (drag and drop or export/import). This sucks, compared to me simply selecting one or several rows in Excel by hitting CTRL+C and then CTRL+V. Simple copying of test cases is extremely important to me.

Test States

1. Zephyr uses the states of Pass, Failure, Unexecuted, WiP, and Blocked. These are good test states, but it lacks the two states that are geared a bit for the Test Lead: Duplicate (DUP), and Not-Applicable (NA). These are important states to me, primarily because a test case written by a test lead should never be deleted but sometimes they can be redundant across sections of test cases (needing DUP) or not-applicable because the requirements have changed, or the requirement needs to be assessed no matter what (needing NA). I don't see a good way to manage these scenarios in Zephyr with the states provided.
2. I'm a big fan of usability, and colors help with that a lot. I don't like the fact that Zephyr doesn't color their test case state (e.g. pass=green, fail=red).

Test Case Library and Templating
The nearest thing I could find to be "Test Case Library like"  is their import/export from file feature. I found that using it was truly cumbersome since I'm used to copying and pasting sets of test cases from one place to another within seconds. If you have or want the ability to hold a library of test cases, or better yet a library of templated test cases then you're going to have to get really crafty with their app infrastructure (e.g. create a project and consider that your library).

Desktop client
In Zephyr, there is a lot of data in a lot of different screens and for the most part that is a very good thing. This was confusing at first, but the more I used and learned the app the more it made sense. The client is the browser with a Flash app running inside it. Working in a Flash app didn't make usability or intuitiveness any easier though. The learning curve for me was a bit steep due to fumbling around with right click context menus in some places and not others to find features (if it didn't exist I'd get the "Flash Settings" context). This is a huge pet peeve of mine and reminded me of working in a poorly written Java GUI.

Metrics
A++. Love them. They're informational and visually appealing!

Requirements
Zephyr requirements traceability is lacking. You can attach a requirement document to a test case but can't point to a specific requirement within that document. I suppose a guy could hijack an existing text input field to create a requirement number or reference in the attached doc (unless you're okay with embedding it within the test case description). This is discouraging if you're looking to tie a test case to a a specific requirement number.

Resource Management
You can assign and schedule test resources to your projects. This is really nice! Currently people/resources can be input into and then managed through Zephyr. I didn't find an evidence of integration Active Directory or LDAP though. This could be a pain if you have many people on the QA team.

Defect Tracking
Zephyr integrates with Bugzilla. Good choice Zephyr! However, if you've customized your Bugzilla interface this feature won't work for you (yet) since the default Bugzilla interface is duplicated inside of Zephyr.

Sharing
Zephyr allows sharing, It even manages test roles: Manager, lead, tester. Roles would be nice in large QA departments. Again this might be a bit more convenient to manage with Active Directory or LDAP integration.

Price
I'm a little discouraged by the price, mostly because they touted it as inexpensive during beta. After release the license model and cost is: "a simple yet flexible licensing model based on monthly subscriptions. Each user license is a low $65/user/month". Let me help you with a few prices for a ONE year subscription: 10 users = $7,800, 100 users = $78,000. That's not quite "almost giving it away!" as they state on their front page.

In a nutshell, I think Zephyr has done a great job with sharing the tester and test lead world with each other and management. However from a test lead perspective I'm a bit disappointed: The way test cases are written, managed, and copied needs to be improved vastly.  Writing test cases in Zephyr is not any easier or better than Mercury Quality Center (which frankly sucks in my opinion). That's a serious problem when 1/4 of a test lead's life is spent writing test cases. Zephyr is new though, give it some time and I think it will shine. In all fairness, Zephyr does a WHOLE lot more than what my spreadsheet is capable of, but I don't think I need that whole lot more, especially when I lose my test case writing convenience.

Everybody's needs, situation, and environment are different though, so go "kick the tires" yourself at:  http://demo.yourzephyr.com

Occasionally people need to access Team Foundation Server source control but they don't have or need Visual Studio or Team System. The good news is that you can access Team Foundation Server without Visual Studio!

Did you know you can access/read TFS source using Attrice's Team Foundation Sidekicks and it's free!?

Did you know you can access/read/write TFS source using Microsoft's standalone application Visual Studio Team System 2008 Team Explorer and it's free?!

Now you do.

Have you ever been in the position where you want to convert an already existing Visual Studio 2005 project to be a Test Project so that you can write and run unit tests? There is no simple way, but there is a hack.

Assuming that you are using a version of Visual Studio that supports Testing (e.g. Team Edition for Software Testers) here is how to do it:

1. Add the following .NET Reference to the project:
   Microsoft.VisualStudio.QualityTools.UnitTestFramework
2. Put in the following using directive in the .cs file that contains your tests:
   using Microsoft.VisualStudio.TestTools.UnitTesting;
3. Create a Test List, this will generate the .vsmdi file (This is the file that opens your Test Manager tab):
   1. Click Test in the top menu
   2. Select Create New Test List:
      
   3. Notice in the Solution Explorer that the Project now is wrapped in a Solution and the .vsmdi file has been created:
      
4. Setup your test configuration file (.testrunconfig):
   1. In Solution Explorer right click the Solution Items folder, select Add, then select New Item
       
   2. In the dialog, highlight Test Run Configuration on the left and select Test Run Configuration on the right side
   3. Name the file accordingly and then click Add:
      
   4. Notice the .testrunconfig file has been added to the solution
      
5. Close Visual Studio
6. Open the project's .csproj file in a text editor.
7. In the top <PropertyGroup> node, change the <ProjectTypeGuids> node to read:
   <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

   If your using VB.NET your GUID should be:
   <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

8. Save and close .csproj file.
9. Open the Solution up (.sln file).
10. You are now ready to write tests.
11. To run your tests:
    1. In Solution Explorer double click the .vsmdi file to open the Test Manager window
    2. Check the test you want to run
    3. Right mouse click the test and select Run Checked Test.

This problem is quite interesting due to the fact that the .vsmdi file and .testrunconfig exist at the Solution level and not the Project level. In the past this interesting fact was a small hurdle for our automation team because we needed to create a Visual Studio template for a Test Project, but Visual Studio only templates Projects and not Solutions (again the .vsmdi & .testrunconfig need to exist at the .sln level). I'll save the explanation of that workaround for another day!

I've scoured the Web for this 3 times now. For your and my benefit I'll store these hard to find links here for future reference...

Visual Studio 2008 Team Edition feature comparison:

http://msdn2.microsoft.com/en-us/vsts2008/products/bb991841.aspx

Visual Studio 2008 Pro and Standard Edition comparison:

http://msdn2.microsoft.com/en-us/vs2008/products/bb980920.aspx

Keep in mind this blog isn't just for you dear reader, it's for my overwhelmed mind too...

The IE team announced what the user agent string for IE 8 will be:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)

This particular one is for Vista. They've decided not to use the 'b' in 'MSIE 8.0b' for the beta version this time due to issues they've encountered in the past with that approach. I've updated my User Agent Switcher import file to contain the string for Internet Explorer 8 on Vista and XP. As always you can get to that file from the right menu under the "My Testing Tools" section: User-Agent Information and Tools. Load up the new file and go see if your browser detection script will work with IE's new string.

I also updated it with FireFox 2.0.12 for Vista.

This one is pretty cool in the fact that you can point it towards a Web site/Silverlight application and it will allow you to peruse the plugin using i'ts XAML Explorer as well as change the properties and and actual XAML on the fly. Seeing tools like this tells me that automating a Silverlight application is easily doable. Flash on the other hand...an automation nightmare. I wish it would go away.

Here is the current feature list:

• Convenient XAML object explorer
• XAML object property grid allows for getting and settings property values
• Regeneration of the XAML xml based on the object model
• Statistics of used objects
• Tracing facility for debugging Silverlight 1.0 applications

Download Silverlight Spy.

Silverlight Documentation.

Max has some interesting points about the long ago chosen language choice for the bug tracking tool Bugzilla: Perl. What is the popular Bugzilla's future, due to the limitations of the Perl? It sounds like they are thinking about a rewrite. Can an open source product survive a transition of that magnitude? This will be interesting to see how it pans out. Open source failed me once, I hope it doesn't fail the testing community.

I've decided to dedicate a spot on the blog for the user-agent information I've been providing over the last few years. From now on you can reference this post from the right navigation under the tools section.

Here are four very good lists and classifications of user-agents:

1. http://www.user-agents.org/index.shtml
2. http://www.botsvsbrowsers.com/
3. http://www.tnl.net/ua
4. http://www.pgts.com.au/pgtsj/pgtsj0212d.html

Here is my huge user-agent XML import file for use with the User Agent Switcher Firefox add-on: AgentStrings20080223.xml (19.67 KB). Read here how you can use this tool for testing. The current file contains the following user-agents for your browser spoofing pleasure:

Internet Explorer
MSIE 8 (Win Vista)
MSIE 8 (Win XP)
MSIE 7 (Win Vista)
MSIE 7 (Win XP)
MSIE 7 (Win Storage Server)
MSIE 6 (Win XP)
MSIE 5.5 (Win 2000)
MSIE 5.5 (Win ME)
MSIE 5.0 (Win 95)
MSIE 4.01 (Win 95)
MSIE 4.0 (Win NT)

FireFox
FireFox 2.0.12 (Win Vista)
FireFox 2.0 (Win XP)
FireFox 1.5.0.6 (Fedora Core 5/Linux)
FireFox 1.5.0.4 (Debian Linux)
FireFox 1.5.0.3 (Mac OSX)
FireFox 1.5.0.1 (Win Vista)
FireFox 1.5 (Win XP)
FireFox 1.4 (Win XP)
FireFox 1.0.7 (Win XP)
FireFox 1.0.6 (Win XP)
FireFox 1.0.5 (SlackWare/Linux)
FireFox 1.0.4 (FreeBSD)
FireFox 1.0 (Win XP)
FireFox 0.9.3 (Win XP)

Opera
Opera 9.1 (Win XP)
Opera 8.0 (Win 2000)
Opera 7.6 (Server 2003)
Opera 7.51 (Win XP)
Opera 7.5 (Win XP)
Opera 7.5 (Win ME)
Opera 4
Opera 3
Opera 2
Opera 1

Netscape
Netscape 8.1(Win XP)
Netscape 8.0.1 (Win XP)
Netscape 7.2 (Win 2000)
Netscape 7.1 (Win 98)
Netscape 4.8 (Win XP)
Netscape 4.76 (Win 98)
Netscape 4.5 (Win 98)
Netscape 4.05 (Win 95)
Netscape 3.01 gold (Win 95)
Netscape 2.02 (Win 95)

Safari
Safari 3.0 (v521)
Safari 2.0 (v418.9.1)
Safari 2.0 (v412)
Safari 2.0 (v401)
Safari 1.3 (v162)
Safari 1.3 (v158)
Safari 1.3 (v146)
Safari 1.2(v125.8; OSX 10.3.4)
Safari 1.1 (v100.1; OSX 10.3.2)
Safari 1.0 (v85)
Safari 1.0 (v74 beta 2)
Safari 1.0 (v73 beta 2)
Safari 1.0 (v64 beta)
Safari 1.0 (v51 update 1.0 beta)
Safari 1.0(v48 public beta)

AOL
AOL 9.0/MSIE 6.0 (Win XP)
AOL 8.0/MSIE 6.0 (Win XP)
AOL 7.0/MSIE 5.5 (Win 98)
AOL 6.0/MSIE 5.5 (Win 98)
AOL 5.0/MSIE 5.0 (Win 98)
AOL 5.0 (OS X)
AOL 5.0 (Mac PPC)
AOL 4.0/MSIE 3.02 (Win 95)
AOL 4.0/MSIE 4.01 (Win 95)
AOL 4.0 (Mac PPC)
AOL 4.0 (Mac 68k)
AOL 4.0/MSIE 3.0(Win 3.1)
AOL 3.0/MSIE 3.02 (Win 95)
AOL 3.0/MSIE 4.01 (Win 95)

Mozilla
Mozilla 1.7.8 (Linux)

Spiders/Bots
Googlebot 2.1 (New version)
Msnbot 1.0
Msnbot 1.0 Beta (v0.11)
Yahoo Slurp
Ask Jeeves/Teoma

RSS Readers
NewsGator 2.5 (Win XP)
Feed Demon 1.6 (Win XP)
Bloglines 2.1
SharpReader 0.9.6.0 (Win XP)

WAP Devices
Nokia 6630
Nokia 6100
BlackBerry 8703
iPhone (3.0 Mobile)

Toolbars
Google Toolbar 4.0
Hotbar 4.4.2
AskBar 3.0

Other
iTunes 4.2 (OS X 10.2)
iTunes 4.7 (OS X 10.2)
iTunes 7.1.1 (Mac PPC)
Zune 2.0
Facebook

Back in 2006 I wrote the article The $60 Web Testing Toolbox with the intent of providing a cheap set of Web testing tools for testers. Most of the tools were mentioned because they provided me the most bang for the buck, but the reality is that good testing tools sometimes cost money. I wanted to share with you my complete Web Testing Toolbox minus the cheap factor. Developing software is not cheap so why does testing of it have to be? For the most part, these tools are inexpensive and have proven to provide great value. In my experience, each and every one provides enough value to justify the cost. Keep in mind I've pretty much lived in a .NET environment since 2001, so some of the things in my list tend to lean in that direction.

I'll keep this list updated as I see fit. I've provided a permanent link in the right navigation for permanent and quick access.

Also, If you're shopping for testing tools and you want an extensive list of what is out there, check out ApTest's tools section.