Web-site layout, one of the many things that can keep a tester busy. Uhm...overwhelmed? So many browsers so little time... Wouldn't it be nice if you, the mighty tester, could just review a butt-load of screen-shots of your Web application in multiple browsers on multiple platforms to make sure there are no layout issues?  Wouldn't that be quick and efficient?

I've got 2 semi-solutions for you (keep in mind I've done little with both, so forgive an misinformation):

Litmus
"We've felt the pain of getting website designs to work correctly across different browsers. Not to mention designing email newsletters that work on all email clients. Litmus makes compatibility testing easier. Litmus is lightning-fast, reliable and affordable. It's relied upon by thousands of smart freelancers and switched-on agencies; as well as big companies like Yahoo!, Facebook and eBay."

The FREE part of Litmus: Screen-shots of your site in IE 7 and FireFox 2.

The $ part of Litmus: Pay $24 a month to get 23 browsers and 14 email clients.

BrowserShots
"Browsershots makes screenshots of your web design in different browsers. It is a free open-source online service created by Johann C. Rocholl. When you submit your web address, it will be added to the job queue. A number of distributed computers will open your website in their browser. Then they will make screenshots and upload them to the central server here."

The FREE part of BrowserShots: 70 browsers on various platforms! Submissions get dumped to a queue for processing.

The $ part of BrowserShots: Pay $15 a month to get priority processing.

Both of these appear to be good services that can provide quick insight to layout problems in your site. However, as far as I can tell the two big limitations are:

• You are limited to pages that you can navigate to via URL, which rips the grandiose dream of having a screen-shot for every page in your website (pages that require form post or special conditions to get to are not going to happen). However, Litmus does provide a step in the right direction with it's functionality for authentication.
• Your site must be exposed to the Web, doing little for internal Dev and QA project cycles.

I have a dream...

• I want to screen-shot any page in my website (requires a decent engine that will allow me to get to the various pages in my site).
• I want to screen-shot my site that is not yet published for the world (requires the service to exist within my local network).
• Once I've approved an ideal layout screen-shot I want the software to determine and tell me if the other screenshots are worth looking at (by doing a statistical image comparison with a predefined pass/fail threshold).
• I want to provide basic wire-frame definitions and have software determine if my screen-shots are within reason (by analyzing elements in the DOM and browser dimensions)
• Get rid of screen-shots and do DOM to DOM element width and height comparisons between browsers (Come on, it's a dream, standards compliance for all browsers (another dream) might make it possible?)

Honestly, I think the dream is doable... So many dreams/ideas, so little time.

For those of you in need of doing SQL load testing from Visual Studio 2005 or 2008 there is a new open source project at CodePlex called SQL Load Test. How does SQL Load Test work?

"This tool takes a SQL Profiler trace file and generates a unit test that replays the same sequence of database calls found in the trace file. The unit test is designed to be used in a Visual Studio Load Test. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing."

Get more info and download SQL Load Test here.

Over the last month I've been looking at Zephyr, a test management system that touts itself as "Next Generation". What exactly is Zephyr and what does it have to offer to the testing community?

"Taking a completely realistic approach to how Test Teams work, collaborate and interact with each other in their department and the rest of their world, Zephyr brings together a comprehensive set of features, a really slick UI and Web 2.0 features at a price point that makes it very affordable for all team sizes.

Zephyr is based around the concept of Desktops & Dashboards. Every role in a Test Department has a customized Testing Desktop with relevant applications that allow them to do their jobs faster and better, as they all share data from a centralized repository and communicate via a collaborative backbone. Dashboards are automated and live, keeping the whole company updated on every aspect of testing and product quality."

At a high level Zephyr offers:

• Testing Desktop
• Dashboards
• Metrics & Reporting
• Test Case Repository
• Resource Management
• Project Management
• Release Management
• Test Case Creation
• Test Execution Planning
• Test Execution
• Document Management
• Defect Tracking
• Collaboration
• Import and Export
• User Interface
• Integration

The 20,00 foot view of Zephyr is this (see the related "How Zephyr works" video here):

Now, I've been managing test cases in a Excel spreadsheet for years, a fairly advanced one at that. It utilizes Visual Basic for Applications (VBA) and gets the job done very well for me, several other QA Engineers, and a few managers. So, having to consider a test management system that costs $$ it a hard to swallow...If it's not broke, don't fix it right? With that eating at me, when looking at Zephyr I decided to compare it to what I have and currently use. Let me tell you what my KISS test management system consists of:

• Quick and easy test case writing
• Consistent test plan and case format
• Reusable test case library that contains commonly used test cases
• Brief but technical test case writing format
• Test case state statistics by section and total
• Testing summary for all testing sections/worksheets
• Test case trends via charts
• Automated coloring of test case status for quick visual reference
• Automated test case to build mapping
• Test case to defect mapping
• Simultaneously sharing between multiple testers
• Tester assignment by section/worksheet

Granted, it's not perfect, but again it works very well and people really like it. With my testing world as the level set, let's jump into the good, the bad, and the things to think about if your considering Zephyr:

Tester Assignment
You can specify users to run specific test cases or whole sections. Very nice!

Copying
I can drag and drop individual cases from one folder to another, but I can't figure out for the life of me how to drag sub-folders of test cases into another folder (they move not copy). I also was unable to successfully import a previous export. If indeed this is possible it's not easy or intuitive (drag and drop or export/import). This sucks, compared to me simply selecting one or several rows in Excel by hitting CTRL+C and then CTRL+V. Simple copying of test cases is extremely important to me.

Test States

1. Zephyr uses the states of Pass, Failure, Unexecuted, WiP, and Blocked. These are good test states, but it lacks the two states that are geared a bit for the Test Lead: Duplicate (DUP), and Not-Applicable (NA). These are important states to me, primarily because a test case written by a test lead should never be deleted but sometimes they can be redundant across sections of test cases (needing DUP) or not-applicable because the requirements have changed, or the requirement needs to be assessed no matter what (needing NA). I don't see a good way to manage these scenarios in Zephyr with the states provided.
2. I'm a big fan of usability, and colors help with that a lot. I don't like the fact that Zephyr doesn't color their test case state (e.g. pass=green, fail=red).

Test Case Library and Templating
The nearest thing I could find to be "Test Case Library like"  is their import/export from file feature. I found that using it was truly cumbersome since I'm used to copying and pasting sets of test cases from one place to another within seconds. If you have or want the ability to hold a library of test cases, or better yet a library of templated test cases then you're going to have to get really crafty with their app infrastructure (e.g. create a project and consider that your library).

Desktop client
In Zephyr, there is a lot of data in a lot of different screens and for the most part that is a very good thing. This was confusing at first, but the more I used and learned the app the more it made sense. The client is the browser with a Flash app running inside it. Working in a Flash app didn't make usability or intuitiveness any easier though. The learning curve for me was a bit steep due to fumbling around with right click context menus in some places and not others to find features (if it didn't exist I'd get the "Flash Settings" context). This is a huge pet peeve of mine and reminded me of working in a poorly written Java GUI.

Metrics
A++. Love them. They're informational and visually appealing!

Requirements
Zephyr requirements traceability is lacking. You can attach a requirement document to a test case but can't point to a specific requirement within that document. I suppose a guy could hijack an existing text input field to create a requirement number or reference in the attached doc (unless you're okay with embedding it within the test case description). This is discouraging if you're looking to tie a test case to a a specific requirement number.

Resource Management
You can assign and schedule test resources to your projects. This is really nice! Currently people/resources can be input into and then managed through Zephyr. I didn't find an evidence of integration Active Directory or LDAP though. This could be a pain if you have many people on the QA team.

Defect Tracking
Zephyr integrates with Bugzilla. Good choice Zephyr! However, if you've customized your Bugzilla interface this feature won't work for you (yet) since the default Bugzilla interface is duplicated inside of Zephyr.

Sharing
Zephyr allows sharing, It even manages test roles: Manager, lead, tester. Roles would be nice in large QA departments. Again this might be a bit more convenient to manage with Active Directory or LDAP integration.

Price
I'm a little discouraged by the price, mostly because they touted it as inexpensive during beta. After release the license model and cost is: "a simple yet flexible licensing model based on monthly subscriptions. Each user license is a low $65/user/month". Let me help you with a few prices for a ONE year subscription: 10 users = $7,800, 100 users = $78,000. That's not quite "almost giving it away!" as they state on their front page.

In a nutshell, I think Zephyr has done a great job with sharing the tester and test lead world with each other and management. However from a test lead perspective I'm a bit disappointed: The way test cases are written, managed, and copied needs to be improved vastly.  Writing test cases in Zephyr is not any easier or better than Mercury Quality Center (which frankly sucks in my opinion). That's a serious problem when 1/4 of a test lead's life is spent writing test cases. Zephyr is new though, give it some time and I think it will shine. In all fairness, Zephyr does a WHOLE lot more than what my spreadsheet is capable of, but I don't think I need that whole lot more, especially when I lose my test case writing convenience.

Everybody's needs, situation, and environment are different though, so go "kick the tires" yourself at:  http://demo.yourzephyr.com

Occasionally people need to access Team Foundation Server source control but they don't have or need Visual Studio or Team System. The good news is that you can access Team Foundation Server without Visual Studio!

Did you know you can access/read TFS source using Attrice's Team Foundation Sidekicks and it's free!?

Did you know you can access/read/write TFS source using Microsoft's standalone application Visual Studio Team System 2008 Team Explorer and it's free?!

Now you do.

Have you ever been in the position where you want to convert an already existing Visual Studio 2005 project to be a Test Project so that you can write and run unit tests? There is no simple way, but there is a hack.

Assuming that you are using a version of Visual Studio that supports Testing (e.g. Team Edition for Software Testers) here is how to do it:

1. Add the following .NET Reference to the project:
   Microsoft.VisualStudio.QualityTools.UnitTestFramework
2. Put in the following using directive in the .cs file that contains your tests:
   using Microsoft.VisualStudio.TestTools.UnitTesting;
3. Create a Test List, this will generate the .vsmdi file (This is the file that opens your Test Manager tab):
   1. Click Test in the top menu
   2. Select Create New Test List:
      
   3. Notice in the Solution Explorer that the Project now is wrapped in a Solution and the .vsmdi file has been created:
      
4. Setup your test configuration file (.testrunconfig):
   1. In Solution Explorer right click the Solution Items folder, select Add, then select New Item
       
   2. In the dialog, highlight Test Run Configuration on the left and select Test Run Configuration on the right side
   3. Name the file accordingly and then click Add:
      
   4. Notice the .testrunconfig file has been added to the solution
      
5. Close Visual Studio
6. Open the project's .csproj file in a text editor.
7. In the top <PropertyGroup> node, change the <ProjectTypeGuids> node to read:
   <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

   If your using VB.NET your GUID should be:
   <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

8. Save and close .csproj file.
9. Open the Solution up (.sln file).
10. You are now ready to write tests.
11. To run your tests:
    1. In Solution Explorer double click the .vsmdi file to open the Test Manager window
    2. Check the test you want to run
    3. Right mouse click the test and select Run Checked Test.

This problem is quite interesting due to the fact that the .vsmdi file and .testrunconfig exist at the Solution level and not the Project level. In the past this interesting fact was a small hurdle for our automation team because we needed to create a Visual Studio template for a Test Project, but Visual Studio only templates Projects and not Solutions (again the .vsmdi & .testrunconfig need to exist at the .sln level). I'll save the explanation of that workaround for another day!

I've scoured the Web for this 3 times now. For your and my benefit I'll store these hard to find links here for future reference...

Visual Studio 2008 Team Edition feature comparison:

http://msdn2.microsoft.com/en-us/vsts2008/products/bb991841.aspx

Visual Studio 2008 Pro and Standard Edition comparison:

http://msdn2.microsoft.com/en-us/vs2008/products/bb980920.aspx

Keep in mind this blog isn't just for you dear reader, it's for my overwhelmed mind too...

The IE team announced what the user agent string for IE 8 will be:

Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)

This particular one is for Vista. They've decided not to use the 'b' in 'MSIE 8.0b' for the beta version this time due to issues they've encountered in the past with that approach. I've updated my User Agent Switcher import file to contain the string for Internet Explorer 8 on Vista and XP. As always you can get to that file from the right menu under the "My Testing Tools" section: User-Agent Information and Tools. Load up the new file and go see if your browser detection script will work with IE's new string.

I also updated it with FireFox 2.0.12 for Vista.

This one is pretty cool in the fact that you can point it towards a Web site/Silverlight application and it will allow you to peruse the plugin using i'ts XAML Explorer as well as change the properties and and actual XAML on the fly. Seeing tools like this tells me that automating a Silverlight application is easily doable. Flash on the other hand...an automation nightmare. I wish it would go away.

Here is the current feature list:

• Convenient XAML object explorer
• XAML object property grid allows for getting and settings property values
• Regeneration of the XAML xml based on the object model
• Statistics of used objects
• Tracing facility for debugging Silverlight 1.0 applications

Download Silverlight Spy.

Silverlight Documentation.

Max has some interesting points about the long ago chosen language choice for the bug tracking tool Bugzilla: Perl. What is the popular Bugzilla's future, due to the limitations of the Perl? It sounds like they are thinking about a rewrite. Can an open source product survive a transition of that magnitude? This will be interesting to see how it pans out. Open source failed me once, I hope it doesn't fail the testing community.

I've decided to dedicate a spot on the blog for the user-agent information I've been providing over the last few years. From now on you can reference this post from the right navigation under the tools section.

Here are four very good lists and classifications of user-agents:

1. http://www.user-agents.org/index.shtml
2. http://www.botsvsbrowsers.com/
3. http://www.tnl.net/ua
4. http://www.pgts.com.au/pgtsj/pgtsj0212d.html

Here is my huge user-agent XML import file for use with the User Agent Switcher Firefox add-on: AgentStrings20080223.xml (19.67 KB). Read here how you can use this tool for testing. The current file contains the following user-agents for your browser spoofing pleasure:

Internet Explorer
MSIE 8 (Win Vista)
MSIE 8 (Win XP)
MSIE 7 (Win Vista)
MSIE 7 (Win XP)
MSIE 7 (Win Storage Server)
MSIE 6 (Win XP)
MSIE 5.5 (Win 2000)
MSIE 5.5 (Win ME)
MSIE 5.0 (Win 95)
MSIE 4.01 (Win 95)
MSIE 4.0 (Win NT)

FireFox
FireFox 2.0.12 (Win Vista)
FireFox 2.0 (Win XP)
FireFox 1.5.0.6 (Fedora Core 5/Linux)
FireFox 1.5.0.4 (Debian Linux)
FireFox 1.5.0.3 (Mac OSX)
FireFox 1.5.0.1 (Win Vista)
FireFox 1.5 (Win XP)
FireFox 1.4 (Win XP)
FireFox 1.0.7 (Win XP)
FireFox 1.0.6 (Win XP)
FireFox 1.0.5 (SlackWare/Linux)
FireFox 1.0.4 (FreeBSD)
FireFox 1.0 (Win XP)
FireFox 0.9.3 (Win XP)

Opera
Opera 9.1 (Win XP)
Opera 8.0 (Win 2000)
Opera 7.6 (Server 2003)
Opera 7.51 (Win XP)
Opera 7.5 (Win XP)
Opera 7.5 (Win ME)
Opera 4
Opera 3
Opera 2
Opera 1

Netscape
Netscape 8.1(Win XP)
Netscape 8.0.1 (Win XP)
Netscape 7.2 (Win 2000)
Netscape 7.1 (Win 98)
Netscape 4.8 (Win XP)
Netscape 4.76 (Win 98)
Netscape 4.5 (Win 98)
Netscape 4.05 (Win 95)
Netscape 3.01 gold (Win 95)
Netscape 2.02 (Win 95)

Safari
Safari 3.0 (v521)
Safari 2.0 (v418.9.1)
Safari 2.0 (v412)
Safari 2.0 (v401)
Safari 1.3 (v162)
Safari 1.3 (v158)
Safari 1.3 (v146)
Safari 1.2(v125.8; OSX 10.3.4)
Safari 1.1 (v100.1; OSX 10.3.2)
Safari 1.0 (v85)
Safari 1.0 (v74 beta 2)
Safari 1.0 (v73 beta 2)
Safari 1.0 (v64 beta)
Safari 1.0 (v51 update 1.0 beta)
Safari 1.0(v48 public beta)

AOL
AOL 9.0/MSIE 6.0 (Win XP)
AOL 8.0/MSIE 6.0 (Win XP)
AOL 7.0/MSIE 5.5 (Win 98)
AOL 6.0/MSIE 5.5 (Win 98)
AOL 5.0/MSIE 5.0 (Win 98)
AOL 5.0 (OS X)
AOL 5.0 (Mac PPC)
AOL 4.0/MSIE 3.02 (Win 95)
AOL 4.0/MSIE 4.01 (Win 95)
AOL 4.0 (Mac PPC)
AOL 4.0 (Mac 68k)
AOL 4.0/MSIE 3.0(Win 3.1)
AOL 3.0/MSIE 3.02 (Win 95)
AOL 3.0/MSIE 4.01 (Win 95)

Mozilla
Mozilla 1.7.8 (Linux)

Spiders/Bots
Googlebot 2.1 (New version)
Msnbot 1.0
Msnbot 1.0 Beta (v0.11)
Yahoo Slurp
Ask Jeeves/Teoma

RSS Readers
NewsGator 2.5 (Win XP)
Feed Demon 1.6 (Win XP)
Bloglines 2.1
SharpReader 0.9.6.0 (Win XP)

WAP Devices
Nokia 6630
Nokia 6100
BlackBerry 8703
iPhone (3.0 Mobile)

Toolbars
Google Toolbar 4.0
Hotbar 4.4.2
AskBar 3.0

Other
iTunes 4.2 (OS X 10.2)
iTunes 4.7 (OS X 10.2)
iTunes 7.1.1 (Mac PPC)
Zune 2.0
Facebook

Back in 2006 I wrote the article The $60 Web Testing Toolbox with the intent of providing a cheap set of Web testing tools for testers. Most of the tools were mentioned because they provided me the most bang for the buck, but the reality is that good testing tools sometimes cost money. I wanted to share with you my complete Web Testing Toolbox minus the cheap factor. Developing software is not cheap so why does testing of it have to be? For the most part, these tools are inexpensive and have proven to provide great value. In my experience, each and every one provides enough value to justify the cost. Keep in mind I've pretty much lived in a .NET environment since 2001, so some of the things in my list tend to lean in that direction.

I'll keep this list updated as I see fit. I've provided a permanent link in the right navigation for permanent and quick access.

Also, If you're shopping for testing tools and you want an extensive list of what is out there, check out ApTest's tools section.