Overview
In this screencast I walk you through doing basic browser setup and automation using a free technology stack: Visual Studio 2010 Express, the WebAii Testing Framework from Telerik, C#, and MbUnit/Gallio. Why this stack you ask? Well, what I'm showing you is a FREE, non-trial, stepping stone to help you do a proof a concept with some leading edge tools in .NET without forking out the money.

One thing that we struggle with in our department is helping those who are interested in doing automation explore the possibility while aligning with technologies the development team is using. The problem is the initial cost of Microsoft's Visual Studio and Telerik's WebUi Test Studio (which uses the WebAii API) is far to much for somebody that is just tinkering (greater than $5000). This technology stack enables you to tinker, but keep in mind you will be missing out on a LOT of features and productivity enablers that come with the product that you pay for. For example with WebUi Test Studio Developer Edition you can get Visual Studio integration which gives recording functionality (a huge productivity enabler) and with Visual Studio Pro the ability to use WebUI project templates, advanced debugging, etc. Ideally, once you've proven out your POC for management you would be able to solicit for the funds for the full product, which in turn would open up another world of collaboration and productivity.

Download

	WMV			iPod/iTunes
	Zune			MP3

The Software Testing Club has done it again! Their quarterly "print", has been redone and now we have "The Testing Planet" as a newspaper! I just got mine in the mail the other day, and I have to say it's pretty dang unique, informative and fun. You too can buy your own Printed Copy, but if you can't find it within yourself to go old school and read the printed paper, you can download it for free here. You should probably buy a copy for safe keeping though, because it contains two of the worlds best QA comics: Cartoon Tester and Do Loop Until 0. Which makes the paper INVALUABLE!

Go Daddy's Hosting R&D department is looking to hire a Software Automation Engineer/SDET. Check out the job posting here. This is an awesome opportunity my QA friend. Let me tell you why:

1. You'll be working with me! Uhmm, why so quiet?. not impressed? Never mind I said that then. How about you'll be working at the same company as Danica Patrick then?
2. You'll be working with cutting edge technology.
3. Our process is agile (we practice Kanban).
4. We work on "cloud computing" products.
5. The opportunity to expand your test and development skills is huge.
6. The team is tight knit, super smart, we work hard, and yet have fun.
7. The team is passionate about our products.
8. As an Automation Engineer you aren't trapped into using one set of test tools or a language. We use what works best and makes sense for the situation.
9. You'll be in the position to mentor other engineers in the art of automation.
10. You'll be surrounded by other Automation Engineers which gives you the opportunity to collaborate with equally minded people.
11. Oh yeah, and Go Daddy in general is awesome place to work, check out the Go Daddy careers page for more info.

You know you want to! Apply here.

Fellow tester and cartoonist Andy Glover showed up on the SQA testing cartoon scene earlier this year with his comic strip Cartoon Tester. Andy does a great job in recognizing, mocking, and illustrating the classic tester and developer relationship as well as the general testers' life. When I discovered and read his comic for the first time in the first Software Testing Club magazine I found myself laughing out loud.

The sad, sad truth about software development. It's hilarious! Make sure and subscribe to Andy's comic here.

Watir Podcast #32 is out and in this episode Zeljko and Gregg have me as a guest! In #32, we spend some time talking about how our Hosting Team at GoDaddy uses Watir for website automation, the supporting framework and patterns, and much more. When your ears have a few spare minutes you can get the podcast at WatirPodcast.com and TestingPodcast.com.

Being able to to run individual tests repeatedly and also being able to run them consecutively is a serious challenge when it comes to automation. I think all automation engineers want it, but some fall prey to not having it due to time constraints, setup and teardown complexity, or system access limitations.

What good is automation when it can't be ran repeatedly, further more running them all at once? Seems like a dumb question, but I've seen plenty of non-repeatable tests or suites written by others, and occasionally have fallen prey myself.

How do I ensure repeatability?

• When planning the automation time budget, I allocate at least 15% for building infrastructure. Can't get the time? Fight for it, as an automation engineer it's your job. You have to sell the fact that if you don't have the time to build in the infrastructure for repeatability your tests will be less useful, take longer to setup due to manual intervention, and will eventually crumble into uselessness over time because manual intervention is not something you can easily hand off to another automation engineer or developer.  
• I start building in repeatability with test #1. This test can often take the longest to write because I'm often times building in the infrastructure needed to run all my tests repeatedly (reusable functions, system access functions (db connections, etc), setup and teardown functions). 
• When the test is done being developed,  I run the test, I run it again, I run it again. repeat and rinse until it passes every time.

How do I ensure I can run all my tests consecutively?

• I build each test so that it can be ran independently, which means that when I run them all from a list, no matter which order they are in, they will not interfere with each other. This is why extracting common code into methods, setup and teardown is instrumental; I will reuse it in almost every test.
• I try to run all tests after each new test is developed and completed to make sure the new test plays nice with the others. This isn't always possible if you have long running tests, but if they aren't, I definitely do this.
• I run them all at least once a day.

The first copy of the Software Testing Club magazine (STC) is out! I had a chance to review it before the release and I have to say it's a really cool magazine. It's fun, different, and not so stuffy. SQA talk can get so boring, but STC breaks out of that box. Between the articles written by the community, the artwork, the comic strips, and the smart-ass QA and development quotes on the bottom-left of every other page, I found myself having a great time reading it. I really enjoyed the comics by Andy Glover, and found myself laughing out loud over them.

I'd like to also point out that my Do Loop Until 0 comic is in the magazine. Not necessarily funny, but a quick view of the realities of testers and developers in the software development environment. I'll admit Do Loop Until 0 can be a little deep at times, but if you study the details closely the irony will hit you like  a sledgehammer. The more I do the strip, the more I realize how Spy vs. Spy influenced me as a child.

Take a look yourself here, I really do think you'd enjoy it.

It's on the horizon. a new, unique and fun magazine written by software testers for software testers: STC Magazine. Keep you eyes peeled this weekend for it on the Software Testing Club site. I'm highly confident you're going to like it.

See more development and QA related comics at Code Comics.

Toot your automation horn! "beep-beep!" Or is that a "HONK-HONK!"? Typically people don't know what your up to in your little test automation world if you don't communicate/toot. Communication gets it out there, getting it out there will allow it to spread. Verbally, in status reports, in executive summaries, etc. "What do I toot?", you say?  Toot your success and your failure:

1. Toot: Your test stats:
   • Calculate time saved by running automated tests vs. manually running the tests. Toot the time saved per test run, per week, per month, per year.
   • Automated test case count
   • Test assertion count (often time x4 the number of tests) 
   • Count of and description of defects found
   • Count and description of defects found through early involvement
2. Toot: Your test framework features and value
   • Code reuse
   • Consistency
   • Shared tests
   • Patterns and practices
3. Toot: Your failures:
   • So that other automation engineers don't make the same mistakes
   • To keep things realistic. Positive only is hard to believe!

There is a fine line for tooting automation, "To toot or not to toot?", that is the question. Don't be (too) cocky. For example, a good toot is "Automated regression passed! Now that's nice, the state of the build determined in 2 minutes!". A bad toot: "This automation is so awesome, you guys would be screwed without it!". Don't over toot. Nobody likes an annoying tooter. Toot stats in your status report. Verbally toot once or twice a week to the project/Dev team. Toot your heart out to your fellow automation engineers, they are on the same page.

Zeljko Filipin has put together a site to that encompasses many testing related audio podcasts at TestingPodcast.com. It's amazing to see how audio podcasts have grown in the last year within the testing community. QA and testing voices are literally heard, and that's pretty cool.

Stay tuned to TestingPodcasts.com and you'll be sure to hear my monotone voice in the next month or so. If you're a true fan you've heard it already in my testing screencasts :)

An automation engineer's test automation progress is often a black box to the project team and managers and that is serious "egg on the face" for any automation initiative. One day while automating I started reminiscing about how I used to monitor and report test case status while doing functional testing, and thought to myself "How can I do that with my test automation?". Shortly after. a process and a tool was born, and stats were included in my weekly reports. I also had the ability to provide detailed test descriptions. Now others had insight to my goal, my progress, I could estimate a completion date, and the project team could review my test descriptions looking for voids in coverage. A bonus benefit to tracking status is that multiple automation engineers can work on one project and not accidentally stomp each other. Seems like a no-brainer right?.. But more often than not I see automation engineers working in an automation black box leaving them unaccountable to all.

Here is an example of how I make myself and my test automation accountable:

1. I stub out my test cases when reviewing requirements (the final number is my goal). For example, each test case is usually one method in my automation test suite. One hundred tests equates to 100 methods. I use separate classes to segregate functionality. My method name follow a pattern and are very descriptive, which helps me decipher what they are when they are in large lists and allows for easy alphabetical sorting.
2. When stubbing the tests/method, I write the test description/steps with it's verification points. For example, in the screenshot below, the "Description" attribute contains these details.
3. I track test/method development status. In the example below you can see the various status that I use. Status is the key to monitoring progress!
4. I tie defect ids or agile task numbers to test cases, which makes for easy searching when I'm doing defect regression:
    
   
5. Finally, I use a tool/automation to extract goal, status, and test description: Note that in the above "Stats" screenshot I have a Test Summary "Count" which is my goal, I have a count of the various states, and have a percentage of the various states. "Completed" percentage is my progress towards the goal. I typically take a screenshot of this tab and paste it into my status report.
   
   Note that in the above "Test Details" screenshot, I have a column for Class and Method which allow me to sort by them. Then I have a test "Description", the test "State",  the "Reason" for test blockage, and finally a place for "Comments". This tab is nice for a quick overview of tests, it allows sorting which is nice if you want to, for example, sort by "Blocked". This can also be exported into an Excel spreadsheet. This view is VERY helpful when you end up having hundreds of automated tests, because scrolling through hundreds of lines of code can make things easy to miss or can get confusing.

 
The 5 points made above were done in my .NET test automation environment which uses a custom attribute I created called "TestProgress". The reporting GUI uses reflection to extract the class, method, and attribute details. The example is for .NET but this process and pattern could be used in any language that you may be automating in. For example in a scripting language (e.g. Ruby), you could provide "Test Progress" as a comment  above the method and then use regular expressions to parse the files to create your report. For example, the Test Progress comment could look something like:

2009. What an eventful year. Eventful in my personal life as well as in my SQA career. A good, eventful year.

I didn't blog much in 2009, 17 posts in all, and no topics that were SQA groundbreaking.  Yeah, I'm pretty much ashamed of myself and have watched my blog fall off peoples' radar.If I were to highlight my favorite post it would be my turn from SQA words to illustrations with Do Loop Until Zero. A hit or a miss, I don't know; I don't get comments either way on this blog. But none the less, it's something I enjoy doing. Hopefully you guys will see more of this "comic", if all works out well, it will be in the 1st issue of the new and upcoming Software Testing Club magazine.

Though the blog was quiet, my SQA and testing career wasn't. In the last year I had the ability to start filling a large gap that was present in my testing experience portfolio. Prior to 2009 I had no experience in the Linux world and the technologies that surrounded it. Joining a new group within GoDaddy quickly changed this. In 2009 I did a 180 degree turn from my beloved Windows world and submerged myself in Linux in an effort to test and automate a new, internal product. I was scared to make the jump, mostly because my Windows wisdom would be put to little use, and my lack of Linux knowledge would make me a slower tester and automator. Not so enticing when I really pride myself on speed and efficiency ("Hire me, Hire ME! I'm two testers for the price of one!"). Scared or not it was an awesome opportunity to further my skills, and help a 1.0 product using my experience with agile practices and automation. With the help of an awesome two  man development team, I was able to learn, automate and wade through the following technology highlights in 2009:

Product: A storage system (C, mySQL):

• I used PuTTY as an SSH client to the dev, test and prod environment running CentOS as a flavor of Linux
• I extended developer unit tests and automated API functional and boundary testing with Perl unit testing (Test::Unit)
• I extended PHPUnit to serve as an automation framework for automation of functional tests (use case, boundary, error handling,etc). The framework was named TAEL (Test Automation Ecosystem for Linux).

Product: FTP Server that uses the storage system (Perl, mySQL)

• I automated use cases, and FTP functions using TAEL. FTP functionality was tested using PHP's FTP  library. Validation was done through FTP responses, and mySQL queries.
• I performance tested the FTP server and underlying storage system with Apache JMeter. FTP in JMeter is not very robust, and worse yet forces a connection open, logon and close for every request needed, which is not very realistic. Thankfully it's open source (Java) so I extended it and tweaked it to fit our needs.

Product: User Management Web Service

• I automated use cases, boundaries, etc with TAEL. Validation was done by querying mySQL or parsing the Web Service response using XPATH queries.

Tool: User Experience Monitor

• In an effort to monitor response times on an ongoing basis, I wrote a script that executes basic functionality every 15 minutes, stores the timed results in FTP, where they are picked up and  processed by a chron job that puts the results in a database. Chron takes the results puts them into an XML format which are then viewed in a PHP web page using the chart control XML/SWF charts. We found some very interesting activity and trends through this test/monitor. This turned out to be a very interesting almost real-time QA asset for the team.

Product: REST service

• Automated REST service tests using the Visual Studio 2008 unit testing framework (MSTest, Microsoft.VisualStudio.TestTools.UnitTesting namespace)
• Created a .NET API for the REST service that would serve as an easy to use interface for automated testing as well as others wanting to use the service from .NET.
• Performance tested the REST service using my .NET API and Visual Studio 2008 load testing framework
   

Automation with Ruby: With a department wide goal that everybody must do a little automation, I led them down the path of Ruby/Watir (due to cost, and Ruby being pretty easy to learn). The results are looking pretty good, adoption has gone well and progress is being made. Here are a few details about the framework that I built over a few weekends:

• Uses a pattern that I call "Test, Navigate, Execute, Assert"
• Manages tests with the Ruby library: Test::Unit
• Uses Watir for web page automation
• Web Service automation is done with:  soap4r  & REXML
• MySQL database validation with the gem:  dbd-mysql
• Data driven automation from Excel using Roo

 
Process: Since I've been lucky enough to work with a highly motivated, small team of three, our process needs to be and has been really light. We've been pretty successful at being extremely agile. For project management we followed a scrum-like process for a little over half a year using the tool Target Process, but then moved to a KanBan approach with our own home-grown tool. Recently we moved from the home-grown tool to a trial with Jira, while trying to maintain the project in KanBan style. I have to  say that I really like KanBan, it works particularly well for our team because it is small. When you're as small and tight knit as out team is, we always know what each other is working on, so the more light-weight the better. It seems the largest payoff of these types of process and tools for our team is tracking backlog items as well as giving management insight to what we're up to.

What's in store for me in 2010? Well, I'll likely be working on the same products, but as far as major learning and growth opportunity I'm excited to dive into the awesome new features of Visual Studio 2010 for Testers as well as to learn and use some C++. Now, if I can just convince myself to blog about those experiences as I go.

With help from Gerhard, a QAInsight reader, the User Agent Switcher MONTSTER XML file has been updated to use the new, folder feature. Also, another IPhone user agent has been added as well as one for Chrome. As always, the permalink is here, and can be found in the right navigation under the "My Testing Tools" header, link: "User-Agent Info and Tools".

The recent release of Firefox 3.5 on June 30th appears to be riddled with defects and it's woes have hit the press. The article points out 55 open defects, I went out assessed the mess myself and see 58 defects and 22 enhancements associated with the 3.5 and 3.6a1 milestones. But here is the kicker, all the defects were reported BEFORE the release date of June 30th. Yet they decided to release with the defects anyway. See the summary for yourself here, notice the dates in the "Opened" column for the 3.6a1 list. Mozilla isn't shy about their lack of quality either, on the Mozilla QA site they boldly proclaim the upcoming tester's event "Firefox BugDay - Catch Missed Blocker, Critical, and Major 3.5 bugs!". Jaw dropping key word being: MISSED. Wow! At least we have to admire their transparency right?

Mozilla, these are the things that kill browsers. Firefox has spent a long time acquiring users, and when you have people uninstalling because of issues, well, it's just not good. The picture is bleak. Releases with knowledge of critical, major and and a pile of normal defects is bad software development and makes me question the whole "Community Development" model. Does community development mean to Mozilla "Community QA after a release"?  Let's hope not, I don't want to see another browser disappear off of the Mozilla branch.

MSDN documentation for "Testing the Application" in VSTS 2010:
http://msdn.microsoft.com/en-us/library/ms182409(VS.100).aspx

Video: Functional UI Testing with VSTS 2010
http://channel9.msdn.com/shows/10-4/10-4-Episode-18-Functional-UI-Testing/

How to add a VSTS 2010 coded UI test to a build:
http://blogs.msdn.com/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx

Creating and running a VSTS 2010 coded UI test through a Lab Manager project:
http://blogs.msdn.com/jasonz/archive/2009/05/26/vs2010-tutorial-testing-tutorial-step-2.aspx
http://blogs.msdn.com/mathew_aniyan/archive/2009/05/26/coded-ui-test-from-microsoft-test-lab-manager.aspx

Explanation of the various Test tool names and products:
http://blogs.msdn.com/jasonz/archive/2009/05/12/announcing-microsoft-test-and-lab-manager.aspx

VSTS related blogs:
http://blogs.msdn.com/vstsqualitytools/
http://blogs.msdn.com/amit_chatterjee/ 
http://blogs.msdn.com/mathew_aniyan/

Now that I've updated and found a permanent home for the Browser Compatibility Cheat-Sheet, I thought I'd take the time to share the method to my madness for excavating the rendering engine and JavaScript data from browsers (these are the details in the cheat-sheet). If you're asking yourself "Why do I care Brent?" then I urge you to watch my screencast on Browser Compatibility Testing Risk Analysis. My answer to your question in a nutshell is that you need to care as a developer or tester because fully understanding how browsers work, knowing how similar and different they are, and being able to quickly assess how a feature or change to a Web application will impact your development, testing or regression. I'll leave it at that for now.

It's never really been quick or easy when I mine the data (which seems to be about once a year), but when I reassess the cheat-sheet I typically spend most of the time re-reminding myself of how to gather the info for the various browsers, all the while discovering new sources. Here is my high-level browser info excavating process:

1. Get the info from the browser itself:
   1. Install it
   2. Look at the browser download site, FAQ, & Release Notes
   3. Look at the user-agent string at browserhawk.com
   4. Look at the installed assembly version numbers and text files
   5. Look at the Help/About section
   6. Look at comments in source control and source code (if available)
2. Search the web
   1. Look at keywords on wikipedia (Netscape, V8, WebKit, etc) . Wikipedia has gained a lot of browser data in the last couple years, some of my favorite and most useful links are comparison of layout engines & Browser Timeline
   2. Search by keywords, and look at articles and comments on the Web

And on to a few lower-level details for a few of the most popular browsers:

Internet Explorer

1. Rendering Engine: Trident being the name of the IE rendering engine, it was once believed that the version of Trident matched the the version of the MSHTML.dll found at C:\windows\System32\. But with IE 8.0, the IE team has a reference on their blog that rendering engine in 8.0 is Trident 4.0 which can now be found in the user-agent string. This version conflicts with prior data I had gathered but we'll run with it.  Hopefully, from here on out you can just gather the version from the user-agent string, only time will tell. Here is an example user-agent string for IE8: "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0;)
2. JavaScript Engine: JScript is the JavaScript engine in IE, the file version of JScript.dll found in C:\windows\System32\ will give you version number.

Chrome

1. Rendering Engine: Webkit is the rendering engine for Chrome. To get the rendering engine version, in the browser URL bar just type "About:". The version will follow the "Webkit:" key. Notice that the User Agent contains the same Webkit number, hence the rendering engine version can be gathered from the User Agent string also.
2. JavaScript Engine: V8 is the JavaScript engine for Chrome. The version is listed in the same place as the rendering engine and is the number following the "V8:" key.

FireFox 

1. Rendering Engine: Firefox uses the Gecko rendering engine and the version number can be found in the user-agent string (can be seen by typing "about:" in the URL bar or going to Help->About in the menu). For example in the following string: "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.10) Gecko/2009042316 Firefox/3.0.10", the "rv:1.9.0.10" is the version and the "2009042316" build number. 
2. JavaScript Engine: Firefox's JavaScript engine is SpiderMonkey and I've never found references to version numbers. From what I understand the JavaScript engine is compiled to JS3250.dll in \program files\mozilla firefox (js3250.lib on Linux), but the version number never changes on it even though the date and size do, which makes tracking here fairly useless. I've always gathered ECMA compatibility data from official release notes as well as using the value for "JavaScriptVer " key found when visiting browserhawk.com (click the 'more' link in top-right corner to get this detail).

Safari

1. Rendering Engine: Webkit is the rendering engine in Safari and the version can be found in the user-agent string. For example in the string: "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Safari/528.17" the version number is 528.18. This can also be seen on the webkit.dll file property "Product Version" in Windows (\program files\safari\).
2. JavaScript Engine: With Safari 4.0 and up the JavaScript engine is new and named Nitro. At this time of writing 4.0 is still in beta and I see no official version for Nitro, but hopefully we'll find a reference once it comes out of beta. Regarding past versions for their old engine JavaScriptCore, I've never found version numbers so I only documented the JavaScript ECMA compatibility found at various places on the web as well as the value for "JavaScriptVer " key found when visiting browserhawk.com (click the 'more' link in top-right corner to get this detail).

And there you have it, the method to my madness. If you have differing or better data please feel free to add it in a comment.

This link/post will serve as the official place for the Browser Compatibility Cheat-Sheet and the post can be permanently accessed from the side navigation. Any new updates/links to the cheat-sheet will be put in this post.

What is the Browser Compatibility Cheat-Sheet?

The cheat-sheet is a list of browsers' rendering engine and JavaScript engine versions. This reference provides testers a quick and easy way to view groupings of browsers and their versions to help determine testing redundancy and trim the "browsers to test" list. For further explanation and details watch the screencast: Browser Compatibility Testing Risk Analysis.

Download the latest Browser Compatibility Cheat-Sheet here (zipped Excel file):

BrowserCompatCheatSheet-052509.zip (8.74 KB)

Thanks to co-worker Julio Verano for passing this on:

Here is a 17 minute video from MIX09 on Automated User Interface Testing with Microsoft Visual Studio Team System 2010.

VSTS 2010 looks like it has great potential for testers and developers, but I think Microsoft is still behind in browser automation and functionality when comparing to the great work from Art of Test with WebAii and Design Canvas.  I'm excited that Microsoft is actively pursuing and growing in this space though. It is needed badly! Here is the current proposed platform support for VSTS 2010:

Written over a weekend, Sean Sullivan the CTO of Black Baud lab's created an iPhone browser simulator, and it's FREE!

Basically, what he did was take the Webkit rendering engine from Safari and embed it into a Windows application. Browser requests are being made through Safari Webkit using an iPhone mobile user-agent string. Here is an example of the user-agent string that came into my website while capturing the screenshot at the bottom of this post:

Mozilla/5.0+(iPhone;+U;+CPU+iPhone+OS+2_1+like+Mac+OS+X;+en-us)+AppleWebKit/525.18.1+(KHTML,+like+Gecko)+Version/3.1.1+Mobile/5F136

Setup is as simple as installing Safari, and then running the .exe from Black Baud lab.

Check out the details and a screencast on Black Baud lab's site.

BUT.as always, when it comes to browser compatibility testing there is nothing like testing the real browser on the real platform. Keep these things in mind my dear browser compatibility testing friend:

1. The rendering results will be based on the version of Safari for Windows you have installed and its Webkit version. This will likely not match the version you intend to test on the iPhone. For example, I've installed Safari version 3.2.2 which uses Webkit 525.28.1, the page I'm testing will utilize that version. Make sure you are aware of the iPhone browser version you need to be testing and the Webkit version that comes along with that iPhone browser version. You'll need to install that version of Windows Safari to be in sync with the iPhone.  How do you know which version of Safari/Webkit is running in an iPhone app install? I don't know! I can find no reference or history of versions on the Web. I suppose it could be gathered easy enough by installing each iPhone app version and then either doing a Help->About in Safari? Or gathering the data from the user agent string by going to browserhawk.com. Post a link in the comments if you've seen this data somewhere. 
2. Apparently iPhone Safari 3 is not the same code base as Safari 3! The code base was branched. Changes made in that branch could cause your results to vary between the real iPhone browser and the iPhone browser simulator. From the threads I've read it seems to be more "shell" type features, which lowers your risk if you're just looking for rendering issues.
   

[click to enlarge]

[click to enlarge]

A question from Tobbe Ryber has inspired me to jot down a few things I've been meaning to do in a more extensive blog post for a long time. But since it hasn't happened yet, I figure it probably never will, so you'll have to settle for my abbreviated, half-ass version.

Twenty reasons to use Visual Studio Team System 2008 Test Edition for your software testing automation framework, ESPECIALLY if your development team is using .NET and Visual Studio:

1. You are using the .NET platform which is a set of stable and robust libraries that allow you to do just about anything. Make HTTP requests, make Web Service requests, use COM, make database queries, the list goes on and on. Basically anything your .NET developer is doing you're going to be able to tap into using the same context. Easily.
2. You have the ability to easily make calls into several layers of the application under test using a few lines or code, without duct taping and bailing wiring a bunch of libraries or technologies together. Imagine automating something in the browser, making a call to a Web service and then calling the database to validate your results...All in a few lines of code.
3. There are awesome tools and libraries that are built on .NET that allow you to automate browsers, such as SWEA, WatiN, and HTTPWatch.
4. There is a great library and Visual Studio add-on that allows you to automate multiple browsers (IE 7, 8, FireFox. Safari and Chrome any day now), as well as Silverlight. Best yet, the recorder integrates with the IDE: ArtofTest's WebAii and Design Canvas.
5. Your 'test harness' is built into the IDE, and your tests can also be ran from the command line.
6. The IDE is top notch when it comes to development and debugging (and test development and debugging). I've been using VS for automation since VS 2005, and when I've had to automate in other worlds (e.g. Linux, PHP, and Perl) I honestly feel like I'm working with tools that equate to a chisel and stone tablet.
7. Auto-complete in the IDE is a huge timesaver. Your time spent searching the internet or referring to a library's specifications is far less with auto-complete.
8. Syntax issues with scripting languages (JavaScript, Ruby, etc.) can be a huge waste of time at runtime. If you write a test that runs for minutes, hours, or days it could fail halfway through due to syntax. A compiled language is not going to do this.
9. The Microsoft.VisualStudio.TestTools.UnitTesting namespace is not just for unit testing, it works great for test automation. It feels a lot like nUnit to me.
10. Integrating your tests with development builds is cakewalk. Using the mstest command line, it's easy to have your tests run with a build in TFS or CruiseControl.
11. You have the ability to easily move some of your tests up the chain to run along side of developers' unit tests. By doing this you now have automated acceptance tests, so that releases to QA have higher quality.
12. You are using the same environment/language as your developers. By doing this, you gain:
13. A. The ability to have developers help you with getting over the .NET language or VS IDE learning curve.
14. B. Knowledge and use of the same language and libraries used for development, thus having a greater technical understanding of what you're testing.
15. C. Easily share and discuss your tests with developers because they are familiar with the language you are using.
16. Test results are in an XML format which means that if you want to use something other than VSTS to view results you can easily manage it.
17. The .NET community is huge. Help, technical examples, and issue-workarounds are an Internet search away.
18. Examples are SUPER helpful through MSDN. Training video series such as How Do I and VSTS Learn are a great alternative.
19. VSTS also does load testing.
20. .NET, C#, VB.NET, and Visual Studio experience on your resume are technology skills and buzzwords that lure recruiters.

Seven ways to test browser compatibility, without having to manage your own testing lab:

http://freelancefolder.com/7-fresh-and-simple-ways-to-test-cross-browser-compatibility/

Browser compatibility testing solutions have made some progress in the last year but it's still far from ideal. A while back, I had a dream...

• I want to screen-shot any page in my website (requires a decent, free, scripting engine that will allow me to get to the various pages in my site that can't be accessed via a specific URL (Watir?) ).
• I want to screen-shot my site that is not yet published for the world (requires the service to exist within my internal network).
• Once I've approved an ideal layout screen-shot I want the software to determine and tell me if the other screen-shots are worth looking at (by doing a statistical image comparison with a predefined pass/fail threshold).  Maybe MeerMeer is on the right road to doing this?
• I want to provide basic wire-frame definitions and have software determine if my screen-shots are within reason (by analyzing elements in the DOM and browser dimensions)
• Get rid of screen-shots and do DOM to DOM element width and height comparisons between browsers (Come on, it's a dream, standards compliance for all browsers (another dream) might make it possible?)

I've been using Adobe Photoshop since version 3.0.  Back in the day I had to to insert MANY 3.5" floppy disks to install it on my PC (20 or more I think). Oh, the good old days... Fast forward to 2009. I use Photoshop a LOT. For all kinds of things. Web design, logo design, photography touch up, creative drawing, etc. I LOVE graphic design.

I also love software quality assurance. I've had a desire for nearly two years to express my quality assurance thoughts via a comic. Over those years I've toyed with some ideas, created a few proto-types that included monkeys as testers, and received some great "How-To" advice from Brad Fitzpatrick. Inspired by Brad Fitzpatrick, IKEA assembly instructions, Shadow Culture's Bug Bash, and a TDD poster I think I've come up with something. I'm not sure that we're dealing with a "comic"  because it's not necessarily funny (at least not my first two strips). It's more of an illustration of simple things that occur between a Quality Assurance Engineer and a Software Developer, and yet more complex thoughts or methodologies occurring in the industry. I suppose that it could be more of a graphic storytelling that reveals a sick truth, simple lesson, or moral? I'm a very visual person. Images stick with me better than words. I know I'm not the only one. Through these images and subtle text hints I hope to tease the QA and developer mind and invoke some QA thought.

My new comic is entitled "DO LOOP UNTIL 0", a name that is a software defect itself... In the BASIC language, "DO LOOP UNTIL 0" causes an infinite loop. Infinite loops are just plain bad when it comes to software. :)

For now I'll host it here on QaInsight.net, you can also get to it via DoLoopUntil0.com and DoLoopUntilZero.com. Once it becomes wildly successful and crashes the server in my office closet I'll think about giving it a real dedicated home. Until then, let me know how I'm doing. Your comments are inspiring, good or bad.

Oh, the comic... I'll post the first in the next few days. I'm still tweaking a few brush strokes... err... pixels.

I noticed this on Google this morning at 7:50 AM (GMT 7:00 Arizona):

No matter what I searched for, all results returned returned "This site may harm your computer". Attempting to get to the site through the safe browsing feature resulted in:

I bet the phones were ringing off the hook over at Google about then. All is well now.

public void testMe(int foo){ }

and you wrote a test that called the method like this:

testMe( "poofoo" );

the test would fail at compile time due to static typing (because the method's input parameter type is an int, but we're trying to pass a string)

Thus the roadblock I was describing... There are a lot of tests, poor error handling, AND hidden defects when you try to pass invalid types into inputs in a service. So, about now you are saying..."Who gives a crap Brent, if my service's consumers use .NET they'll compile, get the error and never even get a chance to send a string in where an int should be". There was a day when I thought the same thing (a long time ago), until I consumed a Web Service using a Java application. For example, say you've exposed a .NET Web Service for the world to use... and I come along and consume it with Java, by doing that I can ignore your static/strong typing and send in that string that you weren't prepared for. Strings don't work so well as ints (especially when it's "poofoo", or even better "2147483648"). Most of the time the errors are just plain ugly, making the API or Web Service really hard to work with when trying to do good error handling management (imagine getting "you failed, line 67", but then a code change occurs and now it's line 68), but sometimes things fail on a larger scale and make some really cool defects.

Back to the point...

I'm thinking the new dynamic type in C# 4.0 will allow you to avoid that compile error if you're writing your tests using C#/.NET. Thus, I could send in that string as an int and then see what happens at runtime. That test would look something like this:

dynamic poo = "poofoo" ;
testMe(poo);

Neither confirmed or denied at this point, but if true, I'M REALLY EXCITED. This will fix a major issue that I've had with testing .NET services using .NET.

It's interesting the conversations that are spawned from the issues that come out sending invalid types...

Brent: The error message... "You failed, line 67" is poor and confusing, what did I do wrong?

Developer: It choked when you sent in the wrong type. Don't do that.

Brent: That sucks...so when our customers do the same thing and then call in for help we'll just tell them that?

Developer: Well, uh..no. Okay, I'll put in a little error handling and a descriptive error message.

Brent: Sweet!

Brent: (Evil laugh in head) Hey... Will we support that error message as part of the API? Because, if you make it a message that I, a consumer can rely on, that means my code will depend on it, meaning that changes to the error are breaking changes. Breaking changes must be tracked and documented.

Developer: (Slowly backing away) I CANT HEAR YOU!!!

Interesting... Google Maps has  open sourced their Selenium Test Suite and less than humbly asks you to add to it!

Not only do we test Google's Beta applications...NOW, we can help them automate their applications!

BRILLIANT. Damn these guys are good.

This has been around a while but...

I'll give $20 bucks to the brave QA Engineer who names their son or daughter one of the following:

• <script>alert('xss')</script>
• &lt;script&gt;alert('xss')&lt;/script&gt;
• %3C%73%63%72%69%70%74%3E%61%6C%65%72%74%28%27% 78%73%73%27%29%3C%2F%73%63%72%69%70%74%3E
• PHNjcmlwdD5hbGVydCgneHNzJyk8L3NjcmlwdD4

...poor kid

There is another software testing magazine available for professional testers!...

test experience, boasting 80,000 users (approximately 2 million readers less than QAInsight.net)

Subscribe here: http://www.testingexperience.com/subscribe.php

Read back issues in PDF format here: http://www.testingexperience.com/thanks.php

Don't quit your day job...

In the slim and clean Google Chrome (beta) interface JavaScript issues are not easily detectable unless your page or site is visually broken. Much like FireFox and Safari, a tester must open the "JavaScript console" and keep it in visible view while they test in order to catch JavaScript errors. Chrome's Javascript console is a lot like Safari's, but is a tad worse (see the gotchas at the bottom of this post).

My fellow testers, here is how to get to that JavaScript console and monitor for JavaScript errors while you test:
 

Click the page icon to the right of the URL bar. Select "Developer" and then "JavaScript console" from the menu:

In the top of the JavaScript console window click the "Resources" button:

Start testing. When an error occurs you'll see a red icon with a number in it next to the page that the error occurs:

You'll also see a log of the error in the bottom of the JavaScript console window, if you click the provided link it will take you to the line of code where the error occurs:

Also, if you click the page in the "Resources" section the offending line of JavaScript and the error will be displayed:

As of now, I see two gotchas in Google Chrome Beta when attempting to detect/find JavaScript errors:

1. The "JavaScript Console" reports more than JavaScript issues which makes sorting/distinguishing JavaScript errors from other reported HTML style of formatting errors tough. A cheesy visual helper is to also open up the "Debug JavaScript" window (found in the same Page > Developer menu to the right of the URL bar) and watch for new line items to show up in the log. Those line entries are associated with JavaScript errors, unless..
2. You changes sites/domain, in this case the previously opened JavaScript windows won't display data for the new site. You have to close those instances and re-open them to monitor the new site.  You know that the monitoring has stopped when you see in the "Debug JavaScript" window the following log entry: "lost connection to tab"

Just when you thought you had a handle on your browser compatibility testing, Google gives us Chrome.  What does that mean for you, the infamous browser compatibility tester? Just another browser to test? Yes and no...

Have you seen my previous screencast on Browser Compatibility Testing Risk Analysis? If not, its a worthy watch if you want to get a feel for how the browser generally works and how to trim that browser test list based on a little bit of data and risk analysis. Once that's under your belt, continue on my friend...

Let's dive into the details of Chrome to get a quick grasp on where some new browser compatibility defects for your site may be lurking:

The Layout Engine
What does Google Chrome use for a layout engine? WebKit. The same layout engine that Safari uses. Will it render exactly like Safari? Well, it depends on which version of WebKit that  your installations of Chrome and Safari use. You can determine this quickly by looking at the user-agent string. A quick way to do this is to go to BrowserHawk.com, click "more" in the top right menu, scroll to the bottom of the new page and look for the text "User agent". Here is the Google Chrome user-agent:

Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-US)+AppleWebKit/525.13+(KHTML,+like+Gecko)+Chrome/0.2.149.27+Safari/525.13

As you can see we have a WebKit version of 525.13. Chances are that the latest version of Safari runs WebKit 525 also (again do this by looking at the user-agent of the latest version of Safari). Ignoring the minor version number and focusing on the major version number, if they are the same we can feel comfortable that Chrome and Safari will display layout the same...Meaning they will align objects on the page the same. If you've already tested Safari, chances are you aren't going to find any unique layout defects in Chrome.

The JavaScript Engine
This is where things start to get different. In Google Chrome we have a brand new engine and way of doing JavaScript. Google calls their new engine V8. Safari's engine is JavaScriptCore. Night and day...expect to see differences and potential issues here. When testing make sure to look for JavaScript errors and possible issues with sites that us AJAX (in my next post I'll talk about how to view JavaScript errors in Chrome) .

The Shell
What stands out the most to me in Chrome's "shell" is the fact that tabs, plug-ins, and JavaScript run under there own process. On the surface this looks like an ideal way to manage security and to keep memory in-check, but I would keep an eye on functionality of pop-ups and session management between windows (as in, losing reference where there needs to be reference). Also, notice the clean-cut/different "shell" that Chrome has... This "shell" could also yield potential defects related to printing, or any other page related features that you may find in the various menu bars/icons or "Options" menu.

In summary, if you are asked to test both Safari and Chrome the layout is going to be the same (if the WebKit versions are the same). However JavaScript and the Shell could yield some unique defects.

Web-site layout, one of the many things that can keep a tester busy. Uhm...overwhelmed? So many browsers so little time... Wouldn't it be nice if you, the mighty tester, could just review a butt-load of screen-shots of your Web application in multiple browsers on multiple platforms to make sure there are no layout issues?  Wouldn't that be quick and efficient?

I've got 2 semi-solutions for you (keep in mind I've done little with both, so forgive an misinformation):

Litmus
"We've felt the pain of getting website designs to work correctly across different browsers. Not to mention designing email newsletters that work on all email clients. Litmus makes compatibility testing easier. Litmus is lightning-fast, reliable and affordable. It's relied upon by thousands of smart freelancers and switched-on agencies; as well as big companies like Yahoo!, Facebook and eBay."

The FREE part of Litmus: Screen-shots of your site in IE 7 and FireFox 2.

The $ part of Litmus: Pay $24 a month to get 23 browsers and 14 email clients.

BrowserShots
"Browsershots makes screenshots of your web design in different browsers. It is a free open-source online service created by Johann C. Rocholl. When you submit your web address, it will be added to the job queue. A number of distributed computers will open your website in their browser. Then they will make screenshots and upload them to the central server here."

The FREE part of BrowserShots: 70 browsers on various platforms! Submissions get dumped to a queue for processing.

The $ part of BrowserShots: Pay $15 a month to get priority processing.

Both of these appear to be good services that can provide quick insight to layout problems in your site. However, as far as I can tell the two big limitations are:

• You are limited to pages that you can navigate to via URL, which rips the grandiose dream of having a screen-shot for every page in your website (pages that require form post or special conditions to get to are not going to happen). However, Litmus does provide a step in the right direction with it's functionality for authentication.
• Your site must be exposed to the Web, doing little for internal Dev and QA project cycles.

I have a dream...

• I want to screen-shot any page in my website (requires a decent engine that will allow me to get to the various pages in my site).
• I want to screen-shot my site that is not yet published for the world (requires the service to exist within my local network).
• Once I've approved an ideal layout screen-shot I want the software to determine and tell me if the other screenshots are worth looking at (by doing a statistical image comparison with a predefined pass/fail threshold).
• I want to provide basic wire-frame definitions and have software determine if my screen-shots are within reason (by analyzing elements in the DOM and browser dimensions)
• Get rid of screen-shots and do DOM to DOM element width and height comparisons between browsers (Come on, it's a dream, standards compliance for all browsers (another dream) might make it possible?)

Honestly, I think the dream is doable... So many dreams/ideas, so little time.

For those of you in need of doing SQL load testing from Visual Studio 2005 or 2008 there is a new open source project at CodePlex called SQL Load Test. How does SQL Load Test work?

"This tool takes a SQL Profiler trace file and generates a unit test that replays the same sequence of database calls found in the trace file. The unit test is designed to be used in a Visual Studio Load Test. The code generated is easily modifiable so that data variation can be introduced for the purpose of doing performance testing."

Get more info and download SQL Load Test here.

Things have been a bit quite here at QAInsight for the last few months wouldn't you agree? As always the days and nights are never quite long enough to get all those things I want done.

I have one excuse though.

Actually I have more than one but I'm only going to share one with you:

I've been working undercover with the FBI and Viacom to help parse Google/YouTube logs to obtain logons which relate to IPs, which point to people who are uploading copyrighted content. A mass effort to prepare for the largest bust in digital history.

No... Just kidding.

Really, I'm kidding don't start with that death threat stuff. I've already had three this week.

Apparently the insightful QA advice found on said Software QA blog has increased defect finding and input by QA Engineers across the globe  and developers are angry. Go figure, they want to eliminate me because apparently I'm the ring-leader that has slowed down their development process and release to production.

It's just a blog. I set forth ideas, I didn't do the defect finding. Please spare my life. Please? I have kids.

In fear of my life I'm going to lie low a while.

In the meantime, take some more QA advice from me and go check out Go Daddy's new QA blog BugCrushers.com. They have a QA army of 50 talented individuals. I expect to see some good stuff come from these peeps.

Oh... By the way, if you're a developer and you see posts on BugCrushers.com where the author is "Brent Strange"...

IT'S NOT ME. It's another Brent Strange. Ironically this one does QA too. Go figure.

The alpha version of IETester was recently released and "IETester is a free WebBrowser that allows you to have the rendering and javascript engines of IE8 beta 1, IE7 IE 6 and IE5.5 on Vista and XP" on the same machine.

To good to be true? No, not if you take that sentence literally. These kinds of tools can indeed help you test for "rendering/layout" and "JavaScript" issues, but tend to quickly fall apart when it comes to integration with the OS shell (plug-ins, caching, cookies, modal dialogues, printing etc). IETester is much like MultipleIEs, these types of tools can help when you're in a bind and want to validate something simple that is script or layout related. But to use these tools to conduct all your browser testing versus your dedicated IE systems or VMs will likely bite you in the long run when the issues go beyond layout or JavaScript. IMHO these tools are best left in the hands of the developers to quickly validate layout issues and then QA can follow up with the real thing. It's a good tool to have around but don't bank on it! :)

Download IETester here.

Get a better feel for for how the browser works with my recent screencast: Browser Compatibility Testing Risk Analysis

I just listened to a podcast where Željko Filipin talks with Bret Pettichord about Watir. This is an easy to listen to and informative podcast about Watir. I really admire and appreciate Bret's openness about Watir as he talks about both the good and the bad. If you have 23 minutes give it a listen.

Pex

What is Pex?

"Pex (Program EXploration) is a white-box test generation tool. Given a hand-written parameterized unit test, Pex analyzes the code to determine relevant test inputs fully automatically. The result is a traditional unit test suite with high code coverage. In addition, Pex suggests to the programmer how to fix bugs."

I haven't had a chance to dive in but I'm pretty excited to see how this can speed up production of writing white box tests and/or increase the code coverage on QA's side.

At the end of the day it'd be nice to see this in the developers' hands and not mine. Baby steps though...

Read more about Pex here.

Get an ear full of Pex with a great Hanselminutes Podcast entitled "Pex with Jonathan 'Peli' de Halleux and Nikolai Tillmann".

I hate it.

Unit test this. Unit test that.

Let's put "unit" in our testing framework so we can test "units" cuz that's all ANYBODY is EVER going to test.

Testing outside of the "unit" is forsaken. Thou shalt never test more than a unit. Test more than a unit and ye shall be pelted with stones and crucified.

Let's see... What do we have here for unit test frameworks:

brentUnit (even Brent Strange can have a unit testing framework)
JUnit (notice the word unit)
nUnit (look at me, I test units)
mbUnit (I'm a follower, I can't think outside of the box. Uhm...let's test units.)
csUnit (CHEESE & RICE, this is INSANE)
...

OH HELL...Just go here for the gi-hugey list:

http://en.wikipedia.org/wiki/List_of_unit_testing_frameworks

You back? See what I mean? There's like one-cajillion testing frameworks that are for "Unit Testing".

Unit, unit, unit. It's freakin' beyond dumb.

Tell me...What happens once this developer phase of "Oh, I gotta go write some unit tests, with my unit test framework so my code is better" evolves to tests beyond "unit"? You know, like writing some tests that are "integration" or "functional? It'll happen people.

IT'S HAPPENING PEOPLE.

I, and others have been using "unit testing" frameworks as an automation test harness for years now. All kinds of tests.. Tests that aren't "unit".

Yeah, insane. Unheard of I know. What? You going to crucify me now?

Drop the unit. Get over it. These are "Testing frameworks".

P.S.
Microsoft, nice namespace: Microsoft.VisualStudio.TestTools.UnitTesting

P.P.S
Don't give me any crap about how "unit testing frameworks" are for developers. Good one rocket scientist...Let's create separate frameworks to basically do the same thing. One for Dev, one for QA. Brilliant...

I remember the 1st time I tried to kick-off a "unit test" from Visual Studio 2005. I had to defer to the Web, I couldn't figure it out. A year later and through everyday use of VS 2005 and VS 2008 I've slowly discovered various ways to get those tests rolling. Here are 6 different ways to start a test in Visual Studio 2008:

"Test Tools" Toolbar

Context Menu

Keyboard Shortcuts
Ctrl+R, Ctrl+T: Execute tests in current context
Ctrl+R, Ctrl+C: Execute tests in current test class
Ctrl+R, Ctrl+N: Execute tests in current namespace
Ctrl+R, Ctrl+A: Execute all tests in solution
Ctrl+R, Ctrl+D: Execute the tests in the last test run
Ctrl+R, Ctrl+F: Execute the failed tests of the last test run

Test List Editor

Test Results Pane

"Test" Menu

The votes are in and it's official. Brent Strange has been crowned "The Best Software Tester In the World". Over the course of  the 27 day voting period the software testing world was quite a buzz. Articles, blog posts, forums, and clubs have been consumed with facts, opinions and skeletons for the 7 candidates over the last 4 weeks. QA hubs such as StickyMinds, TestReflections, SQAForums, and SoftwareTestingClub were overtaken with this "best in the world" shakedown as QA professionals and interested developers voiced their opinions. Oddly enough, the recent and parallel presidential campaign even took notice.

An interview in the to be published June edition edition of Better Software Magazine  between Vimh and Brent bring to light why the community's decision on this heralding QA Engineer was not just Internet viral:

Vimh: So Brent, how does it feel being declared the "Best Software Tester In the World"?

Brent: I'm speechless Vimh. I don't deserve this. I test. I do my job. There is no Best Tester In the World. We all bring Quality Assurance to the table in our own unique ways. Honestly, this award should go to the entire Software Quality Assurance community.

Vimh: The people have spoken Brent. You are the "Best Software Tester In the World". How do you think the community came to that decision?

Brent: I'm not sure. I'm guessing that those that I've worked with past and present voted for me, others I assume were followers of my small footprint in the QA community with my blog QAInsight.net. If not acquaintances or followers I suppose it was just viral. 

Vimh: Is it true that you once found and reported 70 defects in one day?

Brent: [Laughing] No... it was 59 defects, but technically 14 of those were enhancement requests.

Vimh: That's a sign of somebody thinking outside of the box I suppose.

Brent: Inside and outside. You've got to be both places at once in this business.

Vimh: Where do you see yourself in 5 years?

Brent: Testing. Writing software to making testing easier and faster.

Vimh: How about 10 years?

Brent: White sand beach with Corona in hand.

Vimh: [Laughing] Sounds like a great goal!

Brent: It's not a goal Vimh. It's destiny. The Internet is gold mine my friend.

Vimh: Agreed. If all goes well I'll see you on that beach with my trophy wife.

Brent: See you there Vimh. I'll see you there...

Vimh: So now that you are empowered with the title "Best Software Tester In the World" do you picture yourself strutting into a developer's cube and saying "listen to be me beeeaaaatch, it's not "function as designed", it's a freakin' defect and you're gonna fix it".

Brent: Uhmm...No.

Vimh: How about: "Hey dumb-ass, you ever heard of unit test?"

Brent: Dude, not funny. We're all on the same team. [Pretty pissed off sounding] Listen, turn this interview around or I'm going to have hang-up.

Vimh: Okay, I apologize....Brent, you've had a few "skeletons in the closet" exposed during the last month.

Brent: Yes, 2 of them to be exact. I have the "never went to college" and "has no testing credentials" monkeys on my back.

Vimh: Do you think that hurt or helped your campaign? 

Brent: [Laughing] Call me a "Rocket Scientist", my "un-educated" guess says "help".

Vimh: Why do you think that is?

Brent: The reality of it is that a QA Engineer's greatest skill is the openness and ability to lean quickly and then apply it towards quality initiatives. You don't need a degree or certificate to do that. Matter of fact, I don't even know of an entity that teaches that skill. I think the SQA community recognizes that. James Bach has brought that to light for us.

Vimh: Interesting. So you're saying that anybody can be a good tester?

Brent: Not "anybody" but "anybody that is open to learn anything, has the ability to lean quickly, and then apply it toward a given task". Software is "anything", it changes daily. SQA Engineers have to stay on their toes and be able to use anything to help prove something.

Vimh: I never thought about it like that. Wow, "anything" is a lot to learn! That's respectful. It's known that you don't care to manage QA. Why?

Brent: I'm a tester AND teacher at heart. Historically a QA manager relentlessly fights the quality assurance battle all day and often times compromises due to budget, schedule and company ignorance. His/her persistence typically pays off in the long run (several years sometimes) though. But...Have you ever taught a fellow QA Engineer something they didn't know and then see that twinkle in their eye when they envision using that lesson in their testing tasks? It's awesome! To me, that's finding more defects by teaching other people to find defects. We BOTH just made software better...together. [Loudly] AT THE END OF THE DAY!

Vimh: Sounds like a pyramid scheme!

Brent: You said it! I'm a walking "QA Amway" and "QA Mary Kay"!

Vimh: [Laughing] Is it true you once said you knew the LDAP protocol to gain a Development team's confidence and respect for testing, but you really didn't know anything about it?

Brent: ONCE? [Chuckling] QA is often looked down upon because they are the 2nd tier information receivers and are expected to magically know everything right off the bat, which is near impossible. A developer can sit, stew, learn, and write code for a new technology for days, weeks, or months and then simply hand it off to QA expecting them to know the same AND know how to test it. To ensure quality and gain respect of that developer I need to  be "open to learn anything, have the ability to lean quickly, and then apply it toward the given task". The Internet is my best friend. So, yes. Quite often "I am clueless, I make sure I project confidence, I go learn the subject after the fact, and then complete the task". There is no harm in that is there?

Vimh: No, I guess not. Especially when you hammer a project with 70 defects in a day.

Brent: [Laughing] Yes, especially when you hammer a project with 59 defects...

Over the last month I've been looking at Zephyr, a test management system that touts itself as "Next Generation". What exactly is Zephyr and what does it have to offer to the testing community?

"Taking a completely realistic approach to how Test Teams work, collaborate and interact with each other in their department and the rest of their world, Zephyr brings together a comprehensive set of features, a really slick UI and Web 2.0 features at a price point that makes it very affordable for all team sizes.

Zephyr is based around the concept of Desktops & Dashboards. Every role in a Test Department has a customized Testing Desktop with relevant applications that allow them to do their jobs faster and better, as they all share data from a centralized repository and communicate via a collaborative backbone. Dashboards are automated and live, keeping the whole company updated on every aspect of testing and product quality."

At a high level Zephyr offers:

• Testing Desktop
• Dashboards
• Metrics & Reporting
• Test Case Repository
• Resource Management
• Project Management
• Release Management
• Test Case Creation
• Test Execution Planning
• Test Execution
• Document Management
• Defect Tracking
• Collaboration
• Import and Export
• User Interface
• Integration

The 20,00 foot view of Zephyr is this (see the related "How Zephyr works" video here):

Now, I've been managing test cases in a Excel spreadsheet for years, a fairly advanced one at that. It utilizes Visual Basic for Applications (VBA) and gets the job done very well for me, several other QA Engineers, and a few managers. So, having to consider a test management system that costs $$ it a hard to swallow...If it's not broke, don't fix it right? With that eating at me, when looking at Zephyr I decided to compare it to what I have and currently use. Let me tell you what my KISS test management system consists of:

• Quick and easy test case writing
• Consistent test plan and case format
• Reusable test case library that contains commonly used test cases
• Brief but technical test case writing format
• Test case state statistics by section and total
• Testing summary for all testing sections/worksheets
• Test case trends via charts
• Automated coloring of test case status for quick visual reference
• Automated test case to build mapping
• Test case to defect mapping
• Simultaneously sharing between multiple testers
• Tester assignment by section/worksheet

Granted, it's not perfect, but again it works very well and people really like it. With my testing world as the level set, let's jump into the good, the bad, and the things to think about if your considering Zephyr:

Tester Assignment
You can specify users to run specific test cases or whole sections. Very nice!

Copying
I can drag and drop individual cases from one folder to another, but I can't figure out for the life of me how to drag sub-folders of test cases into another folder (they move not copy). I also was unable to successfully import a previous export. If indeed this is possible it's not easy or intuitive (drag and drop or export/import). This sucks, compared to me simply selecting one or several rows in Excel by hitting CTRL+C and then CTRL+V. Simple copying of test cases is extremely important to me.

Test States

1. Zephyr uses the states of Pass, Failure, Unexecuted, WiP, and Blocked. These are good test states, but it lacks the two states that are geared a bit for the Test Lead: Duplicate (DUP), and Not-Applicable (NA). These are important states to me, primarily because a test case written by a test lead should never be deleted but sometimes they can be redundant across sections of test cases (needing DUP) or not-applicable because the requirements have changed, or the requirement needs to be assessed no matter what (needing NA). I don't see a good way to manage these scenarios in Zephyr with the states provided.
2. I'm a big fan of usability, and colors help with that a lot. I don't like the fact that Zephyr doesn't color their test case state (e.g. pass=green, fail=red).

Test Case Library and Templating
The nearest thing I could find to be "Test Case Library like"  is their import/export from file feature. I found that using it was truly cumbersome since I'm used to copying and pasting sets of test cases from one place to another within seconds. If you have or want the ability to hold a library of test cases, or better yet a library of templated test cases then you're going to have to get really crafty with their app infrastructure (e.g. create a project and consider that your library).

Desktop client
In Zephyr, there is a lot of data in a lot of different screens and for the most part that is a very good thing. This was confusing at first, but the more I used and learned the app the more it made sense. The client is the browser with a Flash app running inside it. Working in a Flash app didn't make usability or intuitiveness any easier though. The learning curve for me was a bit steep due to fumbling around with right click context menus in some places and not others to find features (if it didn't exist I'd get the "Flash Settings" context). This is a huge pet peeve of mine and reminded me of working in a poorly written Java GUI.

Metrics
A++. Love them. They're informational and visually appealing!

Requirements
Zephyr requirements traceability is lacking. You can attach a requirement document to a test case but can't point to a specific requirement within that document. I suppose a guy could hijack an existing text input field to create a requirement number or reference in the attached doc (unless you're okay with embedding it within the test case description). This is discouraging if you're looking to tie a test case to a a specific requirement number.

Resource Management
You can assign and schedule test resources to your projects. This is really nice! Currently people/resources can be input into and then managed through Zephyr. I didn't find an evidence of integration Active Directory or LDAP though. This could be a pain if you have many people on the QA team.

Defect Tracking
Zephyr integrates with Bugzilla. Good choice Zephyr! However, if you've customized your Bugzilla interface this feature won't work for you (yet) since the default Bugzilla interface is duplicated inside of Zephyr.

Sharing
Zephyr allows sharing, It even manages test roles: Manager, lead, tester. Roles would be nice in large QA departments. Again this might be a bit more convenient to manage with Active Directory or LDAP integration.

Price
I'm a little discouraged by the price, mostly because they touted it as inexpensive during beta. After release the license model and cost is: "a simple yet flexible licensing model based on monthly subscriptions. Each user license is a low $65/user/month". Let me help you with a few prices for a ONE year subscription: 10 users = $7,800, 100 users = $78,000. That's not quite "almost giving it away!" as they state on their front page.

In a nutshell, I think Zephyr has done a great job with sharing the tester and test lead world with each other and management. However from a test lead perspective I'm a bit disappointed: The way test cases are written, managed, and copied needs to be improved vastly.  Writing test cases in Zephyr is not any easier or better than Mercury Quality Center (which frankly sucks in my opinion). That's a serious problem when 1/4 of a test lead's life is spent writing test cases. Zephyr is new though, give it some time and I think it will shine. In all fairness, Zephyr does a WHOLE lot more than what my spreadsheet is capable of, but I don't think I need that whole lot more, especially when I lose my test case writing convenience.

Everybody's needs, situation, and environment are different though, so go "kick the tires" yourself at:  http://demo.yourzephyr.com

ArtOfTest, creators of the browser automation framework WebAii, released a record/playback tool that works from within Visual Studio 2008! The beta release of "Design Canvas 1.0" fills a huge gap in the ArtofTest automation solution. Not only has ArtOfTest provided this great functionality, but this functionality looks really, REALLY cool. See for yourself with their 30 second QuickStart videos. I haven't had time to dig into it myself yet but I'm really excited to do so. From an automation engineer that spends much of his day writing tests in Visual Studio, watching these videos made my mouth water. I'm thinking that ArtOfTest is going to take a lion's share in the automation space in the next few years. They've chosen a great platform (.NET), a great IDE (VS 2008), and are working toward providing test and automation solutions for new technologies such as SilverLight.

Hehe...Funny...'null' is null

James Bach. Man, that guy...

He's my QA hero.

Seriously.

It's not a man-crush though, so don't get all freaked out.

That guy...is able to eloquently match words to the things we struggle with in and as QA. He writes and speaks things that I've struggled to put to words for years.

Recently the TestCrew interviewed James and he spoke of things that I tried to describe recently:

TestCrew: We understand that you're working on a new book about self-education and you will be sharing some ideas on this topic during the June CAST conference. What is the most important thing you've learned about self-education during your career?
James: The single most important thing about that I've learned about self-education is that it isn't an activity I sit down to, as in "now I will educate myself." Rather, it's a way of looking at the world as a puzzle that I must solve, and most of my waking life is an opportunity to discover clues that help me solve it. Walking along a beach can be self-education, or noticing how my son plays a video game. My education comes largely by paying attention while life is going by, and noticing the patterns. For the tester, rapid self-education is essential. We get a half-baked product and must make sense of it. That's fundamentally a learning task.

If you are in QA and you don't follow James Bach then you are just making it hard on yourself. He paves the way. When he speaks...LISTEN. I guarantee you'll learn something.

I mean, really... Look at him, how can you pass up a hero like this?

Jeff Wilcox at Microsoft hat put together a great article entitled "Silverlight Unit Testing" that shows us how to use the newly available unit testing framework that comes with Silverlight 2 (currently in Beta 1). The framework has the ability to test the API as well as run the tests in all the browser that support Silverlight 2!

Check out Jeff's video-cast entitled "walkthrough of the control unit tests" for a quick peek at how this stuff works.

This is really exciting from an automation standpoint. As an automation engineer I always fear new technologies and their hurdles that might make software not so automatable....much like that darn Flash. Here we are almost 10 years later and the best thing we have to "help" automate Flash is iMacros.

We tester's understand the joke in this bumper-sticker oh-so-well don't we? Get yours as a bumper-sticker or T-shirt here. Thanks Jeff for  pointing this out.

Overview
In this screencast learn the art of trimming browsers from a browser compatibility test list by knowing your users, understanding how the browser works, OS & browser facts versus misconceptions, and grouping browsers by common component versions to remove redundancy. This screencast will you help you trim your browser compatibility list and feel confident about it (duration: 44 min. 28 sec.).

Download

	WMV			iPod/iTunes
	Zune			MP3

Presentation Materials
Browser Compat Presentation.zip
(1.62 MB, includes: Browser Compatibility Testing.pptx, Browser Facts Misconceptions and Experience.docx, BrowserCompatCheatSheet.xlsx, Setting Browser Compatibility Testing Expectations.docx)

If you don't have Office 2007 you can get the Office 2007 viewers here for free.

Occasionally people need to access Team Foundation Server source control but they don't have or need Visual Studio or Team System. The good news is that you can access Team Foundation Server without Visual Studio!

Did you know you can access/read TFS source using Attrice's Team Foundation Sidekicks and it's free!?

Did you know you can access/read/write TFS source using Microsoft's standalone application Visual Studio Team System 2008 Team Explorer and it's free?!

Now you do.

I like Mike Cottmeyer's 30 second elevator speech on what agile is basically about. Mike points out 3 key things:

1. A set of engineering best practices that allow for rapid delivery of high-quality software.
2. A project management process that encourages frequent inspection and adaptation.
3. A leadership philosophy that encourages team work and accountability.

Wrong, or right, its a description in 30 seconds that gets your foot in the door. Slamming a 300 page book entitled Agile Software Development on management's desk just doesn't work.

Baby steps people... Changing or improving the software development process doesn't happen overnight. Start with the foot in the door.

I work nearly all day in and around people that are using Visual Studio 2005-2008 and Visual Studio Team Foundation Server, and I keep seeing the issue where a user who once had access to a project in TFS now has read-only or can't even get to TFS. Here are two possible reasons why you now have limited access to a project in Team Foundation Server:

1. The source control setting has been set to Visual Source Safe. Change it back to Visual Studio Team Foundation Server by:
   1. In the top menu navigate to Tools > Options
   2. In the Options window, select the Source Control node on the left side
   3. On the right panel select Visual Studio Team Foundation Server in the dropdown

2. You've changed your network password and the old credentials are cached on your box. Clear cached credentials by:
   1. Navigating to Start and then Run
   2. Type the following text in the run box: control userpasswords2
   3. Press Enter
   4. In the resulting dialog, select the Advanced tab and then click the Manage Passwords button
   5. Select the TFS server name
   6. Click the Remove button
   7. Restart Visual Studio

A long time ago I proved to myself that as a software tester and developer, working with multiple monitors was by far faster and more productive. As a tester the biggest payoff is when I run tests in one monitor while updating the test cases in the other. As a developer the payoff is writing code in one monitor and running the debug app in the other. Though I never had facts to prove this productivity I truly felt and saw it for myself. Fortunately, I've never been in a company that was too poor (or cheap) to provide two monitors so I never had to prove my case. I think the industry is starting to see the light. For example GoDaddy is the first place I've worked for that had 2 monitors as standard issue! Not only that but we are set up to use UltraMon (utility for multi-monitor systems, designed to increase productivity and unlock the full potential of multiple monitors).

If you're a tester stuck with 17" dual monitors or even worse, a single monitor, your evidence is now here... The results are in, the University of Utah says you can be more productive with 2 monitors and the larger the better:

Jeff Atwood, believes these are the actual results of the study.

Hanselminutes recently had a show entitled Quetzal Bradley on Testing after Unit Tests where some very interesting perspectives and points were made by Scott and Quetzal about unit testing, code coverage and testing in general. I've never been a fan of measuring test completeness through code coverage and Quetzal sums up my feelings on that quite well:

"Code coverage can't cover code that's not there"  e.g. lack of error handling, data conditions...

If you're unit testing or writing automated tests be sure to give this show a listen. It'll be worth your time.

Have you ever been in the position where you want to convert an already existing Visual Studio 2005 project to be a Test Project so that you can write and run unit tests? There is no simple way, but there is a hack.

Assuming that you are using a version of Visual Studio that supports Testing (e.g. Team Edition for Software Testers) here is how to do it:

1. Add the following .NET Reference to the project:
   Microsoft.VisualStudio.QualityTools.UnitTestFramework
2. Put in the following using directive in the .cs file that contains your tests:
   using Microsoft.VisualStudio.TestTools.UnitTesting;
3. Create a Test List, this will generate the .vsmdi file (This is the file that opens your Test Manager tab):
   1. Click Test in the top menu
   2. Select Create New Test List:
      
   3. Notice in the Solution Explorer that the Project now is wrapped in a Solution and the .vsmdi file has been created:
      
4. Setup your test configuration file (.testrunconfig):
   1. In Solution Explorer right click the Solution Items folder, select Add, then select New Item
       
   2. In the dialog, highlight Test Run Configuration on the left and select Test Run Configuration on the right side
   3. Name the file accordingly and then click Add:
      
   4. Notice the .testrunconfig file has been added to the solution
      
5. Close Visual Studio
6. Open the project's .csproj file in a text editor.
7. In the top <PropertyGroup> node, change the <ProjectTypeGuids> node to read:
   <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}</ProjectTypeGuids>

   If your using VB.NET your GUID should be:
   <ProjectTypeGuids>{3AC096D0-A1C2-E12C-1390-A8335801FDAB};{F184B08F-C81C-45F6-A57F-5ABD9991F28F}</ProjectTypeGuids>

8. Save and close .csproj file.
9. Open the Solution up (.sln file).
10. You are now ready to write tests.
11. To run your tests:
    1. In Solution Explorer double click the .vsmdi file to open the Test Manager window
    2. Check the test you want to run
    3. Right mouse click the test and select Run Checked Test.

This problem is quite interesting due to the fact that the .vsmdi file and .testrunconfig exist at the Solution level and not the Project level. In the past this interesting fact was a small hurdle for our automation team because we needed to create a Visual Studio template for a Test Project, but Visual Studio only templates Projects and not Solutions (again the .vsmdi & .testrunconfig need to exist at the .sln level). I'll save the explanation of that workaround for another day!

Today I was shown the "Project Alerts" feature in Visual Studio 2008. This cool feature allows you to monitor source control check-ins, failing tests, build completions, and work items in TFS. When one of these events occur you can be notified by email. Monitoring source control closely is very valuable for QA and testers, by doing so you can know the minute something is new or changed and can start building tests or investigating the change.

Here's how to set Project Alerts in Visual Studio 2008:

1. In the Team Explorer window, highlight the project name you want to set alerts for.
2. In the VS 2008 top menu, click Team and then Project Alerts
3. Enable the alert you are interested in and provide your email address in the Send to column.

A past article about uTest has got me thinking.

More than usual that is.

Specifically what has me thinking are these quotes from the article:

"...testers receive feedback and can gain in the rankings, just as with any other promotional network where individual users can establish themselves as authorities."

"Take this developing team model, and it can be applied to a great number of projects that can be rolled out on a variety of websites, allowing uTest to offer specialized niches within its larger network. One way in which this could become particularly helpful is for another one of its upcoming features: a Facebook application. This won’t be a Facebook app like all the others, but a testing platform for app developers. It allows testers to show and their their uTest tester profiles within Facebook (self-promotion at its finest), while also enabling testers to report issues and bugs via the uTest testing platform. This is in fact quite similar to what Facebook itself is taking on with its own Translations Facebook application, which is the social network’s way of better acclimating to the global growth process. Both new developments should work quite well for uTest’s platform, and I think this company has found a great way of letting companies leverage a dedicated, professional social network for quality assurance."

As testers we have a pretty good feeling of where we stand amongst our team at work and other teams within our company. But how do you think your testing skills rate when you compare them with another company, the whole state, the country, and better yet the nation? The notion of sharing your skills with the world is not new to me, but I think uTest and its feature of Facebook application is cutting edge. It's brilliant, scary, good, and bad. A couple years ago my first real leap into the social networking space was with LinkedIn.com, a professional network. LinkedIn.com has done me well. It has proven to be a fantastic resource...Especially if you know how to sell yourself. But beyond LinkedIn, as a Quality Assurance Engineer, I seem to be lost in the sea of other QA professionals. I've attempted to break out a bit with my blog, sharing testing tools I've written, posting advice on forums, joining a software testing club, writing a magazine article, and recently my first public presentation, but still... I am one among many.

It will be very interesting to see the dynamics that uTest ratings and feedback will bring for testers. Will I be able to use a testing rank to gain the edge over another person for a job in the real world? Could my rank be considered a real-time "testing certification"? In my opinion a "testing certification"  is useless without a governing authority, but if I have a good rank and a excellent feedback from 200 people is that a better or more acceptable governing authority? What if it works against me? It only takes a few mistakes, misunderstandings, or one jack-ass to throw off your rank (as seen with XBox Live). It takes a long time to get back into good graces once that happens.

We are definitely seeing a trend for career social networking on the Internet. Recruiters, Human Resources and hiring managers are using the Web to check you out now days (Googling your name, look at your LinkedIn feedback, forum posts etc). As a hiring manager, I wouldn't have a problem with considering your uTest rank and feedback as additional information to help make sure I'm making a good decision about hiring you. Scary huh? Granted, being a good quality assurance engineer isn't all about testing skills, but the more history and data I have on you, the potential hire, the better.

What are you doing to prove your software quality assurance rating in the social networking space?

Performance Wiki: http://performancewiki.com/home.html

Performance Wiki Blog: http://www.performancewiki.com/wordpress/

Get it? A Load of Performance testing info... If you don't, go back to your zone and forget this poor, dry joke ever happened (after you check out the links).

Seriously though, I've never seen so much performance testing information in one spot. Check it out.

I've scoured the Web for this 3 times now. For your and my benefit I'll store these hard to find links here for future reference...

Visual Studio 2008 Team Edition feature comparison:

http://msdn2.microsoft.com/en-us/vsts2008/products/bb991841.aspx

Visual Studio 2008 Pro and Standard Edition comparison:

http://msdn2.microsoft.com/en-us/vs2008/products/bb980920.aspx

Keep in mind this blog isn't just for you dear reader, it's for my overwhelmed mind too...

Oh the monotony of errors we testers endure... If I ran into any of the below errors while testing it definitely would make my day!

What if your company implemented this new captcha and you had to test it:

Following the thinking of Jeff over at Coding Horror...

What software testers do best is learn.

Yes, this captcha is a joke, but if it was real could you test it? Yeah, yeah, I know there are more test cases than solving the equation. But can you solve the equation?

Right now... I can't.

Let me at my favorite resource for 1 minute to an hour (the Internet) and I can.

What software testers do best is learn. Remember that next time you are frustrated with a serious testing challenge. Testing capability is not all about experience, it's also about adaptability, willingness to learn, and the ability to learn quickly. Having those three skills will go a LONG way in the testing world.

"Daily stand-up meetings" now days are attached to the latest and greatest buzzword "agile" in the software development world.  But you, your team, or your company don't have to follow an agile process to benefit from a daily stand-up. Communication is the foundation of any task that involves two or more people and sadly most people are just plain bad at it. Poor, or no communication flat out just doesn't work. Bad communication fails us in so many aspects of our life and we don't even realize it until we start thinking about it and paying attention to it....Work and home. When it fails at home with your spouse what happens? We go to a counselor who advises us to communicate with each other with advice such as "when you get home from work commit 10 minutes to each other (5 minutes each), ask your spouse how their day went, and listen whole-heartedly"... sounds an awful lot like a "stand-up" doesn't it? I'm no psychologist but I'm here to tell you that this lesson is a fundamental building block for success within society. Home or work.

If you're a manager, test lead, tester, woman, or man you need to be communicating...Every day. Don't leave people in the dark. I have personally seen stand-ups work, and WORK EXTREMELY WELL at home and work. Take 2-15 minutes for a stand-up with the people you interact with. You will see an improvement, I have no doubt. Use the patterns of daily stand-up meetings as your guideline.

I did my first presentation for the Phoenix Software Quality Association (PSQA) yesterday evening on "Browser Compatibility Testing Risk Analysis". Those who attended learned:

"The art of trimming browsers from a browser compatibility test list by knowing your users, understanding how the browser works, OS & browser facts versus misconceptions, and grouping browsers by common component versions to remove redundancy."

For those that attended I promised to post my presentation and handouts here on my blog. So here they are:

Packaged Download
Browser Compat Presentation.zip (1.62 MB)

Individual Downloads
Browser Compatibility Testing.pptx (1.65 MB)
Browser Facts Misconceptions and Experience.docx (29.24 KB)
BrowserCompatCheatSheet.xlsx (14.89 KB)
Setting Browser Compatibility Testing Expectations.docx (13.87 KB)

If you don't have Office 2007 you can get the Office 2007 viewers here for free.

I'm thinking about screencasting this while it's fresh in my mind. If I do, it will be posted here within the next few weeks.

PSQA is always looking for presenters or a place to meet. If you can offer up either, the organization would greatly appreciate it (contact info@psqa.org). If you're in the valley, have anything to do with QA, and are not already a PSQA member, go sign up. It's free!

Kudos to Greg for passing on this brilliant code quality metric:

I just signed up as a tester at uTest. What is uTest? Really there are two parts to uTest, people who want their products tested and the testers:

"uTest^TM provides software companies with significant benefits including higher quality software releases, reduction in product release cycle time and overall product time to market while improving overall customer satisfaction. We also provides software companies with significant cost savings including reduction in QA costs by minimizing help desk costs, handling peak QA periods effectively and improving overall quality of released application.

uTest^TM customers gain access to a large, diverse and global community of software testing individuals over our secure testing platform. The uTest^TM Testing Platform provides a hosted infrastructure to manage complete software QA cycles and projects.

Software application testers who are part of the uTest^TM professional testers community are able to test applications in a completely flexible work environment, earn significant additional income and improve technical expertise."

From the tester's perspective, the way it works is that you will get paid for defects that are not duplicate and are verified by the company. It's not apparent what the rates are yet. You can get paid monthly, via a Mastercard debit card from Payoneer, when your balance is greater than $100. I didn't sign up for the money, I signed up because I'm interested to see what testing is like in a large testing like atmosphere. I've only been involved in one other large testing effort for Intel. It was beta testing of hardware and when the testing was complete I got to keep the hardware. The experience was horrible. Mostly because communication sucked and ultimately the product sucked. uTest appears to have a platform built to better manage this type of environment, so I'm excited to see how and if it works well.

I wonder though, since money is involved, will it be more competitive? Will people race to break things and get defects in to make sure theirs isn't a duplicate? I envisioned a few of the typical project, Developer and QA problems and wrote asking how uTest would handle them. Here were my questions:

Bug being duplicate: What if there is a core bug, but another resulting “bug” is caused by the core bug. An example would be an incorrect global variable that displayed itself on 5 different pages. What if reported one defect for each of the 5 pages (resulting in 5 defects), but somebody else reported the core/global variable issue? How and who will manage duplication in a fair way? How will you prevent companies from marking a bug as duplicate as an excuse to avoid paying?

A bug is really a bug: A common scenario that developers & project managers do is mark a bug as a “feature request” when A) the project is overwhelmed and they are trying to lower the defect numbers or B) if the functionality is not documented specifically/a little loosely in the requirements. How will you protect both sides from abuse?

Verification occurred:  At the end of your registration process you state: “If you have encountered a bug in our registration process please let us know by emailing us at bugs@uTest.com, we will credit your uTest account with cash for every non duplicate bug that you submit and we verify”. I’m assuming this is how it will work when testing for other companies too. How will you protect testers from finding/entering defects and not getting paid because the company we are testing for has run out of budget and decided to use the excuse: “not verified, so we’re not paying”?

Usability: Will there be any money for usability suggestions that are turned into future features?

Here is uTest's response:

"One very important point I would like to emphasis: Only the company (the software vendor) is entitled to approve or reject a bug. We do NOT interfere in this process since only the software owner can really identify a real bug or a problem. BUT,  we do offer tester’s a few mechanisms to object a company decision:

1.  A Dispute process -  for complaining about invalid rejection reasons. In that case we will investigate this issue and credit the tester if needed. We would do our best to make sure the testers would get paid for their hard job.

2. Budget -  We do make sure that companies would have enough budget before initiate a new testing assignments, so there is  no way testers would work for companies without approved budget.

3. Tracking invalid behaviors - We track every communication between the company and a tester, so once we’ll notice that a certain company act unfairly with the community we would get involve and make sure the testers is getting what they deserve. The testers community has the power to make a change, once a few tester would report a wrong behavior of a company, we would get involved.

Duplication – As part of our testing platform functionality, we’ve developed a duplication engine in order  to identify a duplicate bug once a new bug is being submitted into the system. We then would notify the tester that is there is a possibility that his new bug is already exist and ask him to make sure he is submitting a different one.  We would also publish a list of known bugs so the testers won’t waist their valuable time finding existing bugs.

Usability – we do plan to offer various methods of credit testers who provide usability feedbacks, this would be in the later stages of the platform."

It sounds like uTest is on the right track. Go check 'em out: http://www.utest.com/

I laughed when I saw this today. Each piece of the pie is so painfully true. I have honestly gone through these phases/woes, over and over and OVER.

After the IE Team posted their intent and work on compliance to standards the debates have begun. Revelations have shown that the standards compliance is really an an option that would be invoked by a developer with a META tag or header using the value of  'X-UA-Compatible'. Now, IE 8 will have 3 modes:

1. “Quirks mode” remains the same, and compatible with current content.
2. “Standards mode” remains the same as IE7, and compatible with current content.
3. If you (the page developer) really want the best standards support IE8 can give, you can get it by inserting a simple <meta> element. Aaron gives more details on this in his article.

There is much debate on how this will impact the Web, user experience, as well as how developers will program. There are interesting thoughts around all of them. What is most intriguing to me is that IE 8 is going to be standards compatible. With news like this, I can now test that a Web application conforms to a standard. The standard is now my guideline for my set of test cases. But really will it be a "set of test cases"? Quite possibly it could be only one: Load the page and validate that the page is rendered using "Best standards mode". But hopefully the test wouldn't even be of my concern, with the right Content Management System, the code will be compiled by a developer and the check will be done by the CMS before release to QA. Either way, conforming to standard lowers the risk of a browser compatibility dramatically.

Another interesting feature in IE 8 is the idea of "Version Targeting". In other words: 

"The idea is that when IE10 loads up my IE7 page, it rewinds itself to act like IE7 did, all those years ago—no matter what changed in the meantime."

What will "Version Targeting", do to my browser compatibility testing list when IE 8 is released? My first impression is that it won't slim my test list at all. For example, if you have a page that is coded for IE 8 and X-UA-Compatible, it doesn't necessarily mean that it will look good in IE 7 or in IE 6 since the code is targeted for IE 8. Using that same Web page (IE 8 and X-UA-Compatible), will the testing list shrink with the release of IE 9? It depends... Will the standards and the compliance to those standards change between IE 8 & 9? How many compliance to standard defects were in IE 8 and fixed in IE 9. If their were few defects and the standards didn't change then the risk is low and you probably wouldn't need to test both IE 8 and 9, just 9.

It's too early to speculate and come up with concrete answers. The IE Team is still working through this stuff with themselves and the active community. But I'm happy to see their attention on the matter.

I swear I put all my browser archive links into one blog post before, but a search tonight proved me wrong. So here they are... These are the sites I've used for years to download old browsers:

FireFox (official)
http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/

Netscape (official)
http://http.netscape.com.edgesuite.net/pub/

Internet Explorer and MANY others (unofficial)
http://browsers.evolt.org/

I noticed the other day that MSDN has a testing section entitled Testing Center. It appears its been out there a while and I somehow missed it (which is amazing when I consider the amount of time I spend on MSDN).

Anyway, there are some interesting things going on out there:

Whiteboard Videos: Decent, short topics. It's hard to read the white board though, which I found really annoying.

Technical Articles: Not too many articles out here, but a good start?

Tester Center Downloads: Three testing tools. Again, a good start.

As an automation engineer that lives mostly in a .NET testing framework I really appreciate MSDN and I'm happy to see a section dedicated to testing. Having MSDN and its abundance of resources as well as video training really makes my job easier. Having these resources available to me also gives me great comfort in my choice to use .NET as the platform for automated testing.

Microsoft, Visual Studio, and MSDN have come a long way over the years. Kudos to Microsoft for making my job easier.

I'll get off my Microsoft soapbox now.

Company: GoDaddy
Job Title: Automation Test Engineer
Location: Scottsdale, AZ

Description
"We are currently seeking to hire an Automation Test Engineer. We are looking for an entry to mid level Test Engineer with a foundation in software engineering and the ability to think through moderate to complex problems. If you thrive in a dynamic environment and enjoy working with cutting edge technology working to define and deliver complete solutions, then we want to see your resume."

Get the complete job description here.

Last month the IE team mentioned that their latest Internet Explorer code (to be IE 8 sometime in the future) renders the "Acid2 Face" tests correctly. This is a large achievement for the team and developers around the world. Browser compatibility testing is going to get easier for us testers (once developers shake their old habits of coding workarounds for browser quirks)! Read the post here.

MSDN in offering more free training! Their new program titled Ramp Up will teach you Visual Studio 2005, C#, and .NET 2.0  after you choose a route based on your current status of Aspiring Developer, Java Developer, Visual Basic 6.0 Developer, or Microsoft Visual Studio 2002/2003 Developer. The online courses and tests are free. Course graduation awards you with major discounts on Microsoft Certified Technology Specialist (MCTS) exams, E-Learning, and Self-Paced Training kits.

If you choose the path of certification or not,  this and other types of training has made becoming a software tester with .NET development skills (SDET) so easy. Being a tester with development skills is the way of the new testing world. Do it.

DO IT.

DO IT!!

I ran into this quote from Albert Einstein today:

In order to be an immaculate member of a flock of sheep, one must above all be a sheep oneself.

I have personally lived this many times over the years in my various careers. I'm going to venture out and say that I've been pretty successful at working my way up to an immaculate member in few cases. I chalk my success up to what I call "working in, and reporting from the trenches", and also, not to ever remove myself from the trenches. I believe that your success amongst your sheep/peers is highly dependant on staying extremely in tune with the trenches, and most importantly the people in the trenches. Once you remove yourself from those trenches, you are not eating the same sheep food, and I consider you a shepherd. Don't get me wrong, shepherds are a good thing, and  shepherds do tune in to the trenches. But they don't always spend 8 hours a day in them because they have shepherd things to do! I try my best to walk the fine line between sheep and shepherd. Am I shepherd in sheep's clothing?

Shepherd, how do you stay in tune with your trenches to gain or maintain an immaculate sheep status?

Overview
This screencast demonstrates how to create a shortcut in the Internet Explorer Links menu bar to the Window's hosts file for quick and easy access while testing (duration: 1 min 58 secs)

Download
WMV
Zune
iPod/iTunes

I've realized the beauty of screencasts at work recently. About a month ago I spent a week and half giving a training presentation 6 times to train the whole QA team at GoDaddy. After presentation #2 I was bored and felt like a programmed robot. Recently, I decided that screencasting might be a better way of training so many people, so I did one. Success or failure is still to be determined, but I really like the idea of being able to show people how to do something rather than explain in it with a ton of words and expect them to venture out on their own and remember what I told them.

Me: Why not carry that thinking into this blog?
You: Great idea Brent!
Me: Do you really think so? Thanks!
You: Don't fail me.
Me: Bite me. You want it or not?

I'm adding a "Testing Screencasts" category. Get ready to do some watching and listening. I'll start small, until I gain some screencasting mojo.

This one is pretty cool in the fact that you can point it towards a Web site/Silverlight application and it will allow you to peruse the plugin using i'ts XAML Explorer as well as change the properties and and actual XAML on the fly. Seeing tools like this tells me that automating a Silverlight application is easily doable. Flash on the other hand...an automation nightmare. I wish it would go away.

Here is the current feature list:

• Convenient XAML object explorer
• XAML object property grid allows for getting and settings property values
• Regeneration of the XAML xml based on the object model
• Statistics of used objects
• Tracing facility for debugging Silverlight 1.0 applications

Download Silverlight Spy.

Silverlight Documentation.

Another year has come and gone for QAInsight.net. To highlight the year, here are 15 popular posts you may have missed in 2007:

Testing Thoughts
Selling quality to the project team
Agile for 30 minutes: Agile defect reporting
Is QA Better at Writing Product Specifications?
Emotional Attachment to Robots, Focus on the Tests Not the Robot
Death By 1000 Cuts, Bad Software Due to Many Low Severity Defects
Keep it Small With a software testing club
Proper use of the sleep method in automation

Testing How To
How to test for database schema differences
How to detect Web service changes between builds
Find installer defects quickly by reviewing key MSI database tables

Testing Tools
Report on dirs, files, and file properties with File Property Reporter
MSI installer testing tool: MSI Database Reader & Reporter
Brent's Web testing toolbox

Fun
How much is an ass-load?
The wonders of caffeinated soap

Max has some interesting points about the long ago chosen language choice for the bug tracking tool Bugzilla: Perl. What is the popular Bugzilla's future, due to the limitations of the Perl? It sounds like they are thinking about a rewrite. Can an open source product survive a transition of that magnitude? This will be interesting to see how it pans out. Open source failed me once, I hope it doesn't fail the testing community.

I've decided to dedicate a spot on the blog for the user-agent information I've been providing over the last few years. From now on you can reference this post from the right navigation under the tools section.

Here are four very good lists and classifications of user-agents:

1. http://www.user-agents.org/index.shtml
2. http://www.botsvsbrowsers.com/
3. http://www.tnl.net/ua
4. http://www.pgts.com.au/pgtsj/pgtsj0212d.html

Here is my huge user-agent XML import file for use with the User Agent Switcher Firefox add-on: AgentStrings20080223.xml (19.67 KB). Read here how you can use this tool for testing. The current file contains the following user-agents for your browser spoofing pleasure:

Internet Explorer
MSIE 8 (Win Vista)
MSIE 8 (Win XP)
MSIE 7 (Win Vista)
MSIE 7 (Win XP)
MSIE 7 (Win Storage Server)
MSIE 6 (Win XP)
MSIE 5.5 (Win 2000)
MSIE 5.5 (Win ME)
MSIE 5.0 (Win 95)
MSIE 4.01 (Win 95)
MSIE 4.0 (Win NT)

FireFox
FireFox 2.0.12 (Win Vista)
FireFox 2.0 (Win XP)
FireFox 1.5.0.6 (Fedora Core 5/Linux)
FireFox 1.5.0.4 (Debian Linux)
FireFox 1.5.0.3 (Mac OSX)
FireFox 1.5.0.1 (Win Vista)
FireFox 1.5 (Win XP)
FireFox 1.4 (Win XP)
FireFox 1.0.7 (Win XP)
FireFox 1.0.6 (Win XP)
FireFox 1.0.5 (SlackWare/Linux)
FireFox 1.0.4 (FreeBSD)
FireFox 1.0 (Win XP)
FireFox 0.9.3 (Win XP)

Opera
Opera 9.1 (Win XP)
Opera 8.0 (Win 2000)
Opera 7.6 (Server 2003)
Opera 7.51 (Win XP)
Opera 7.5 (Win XP)
Opera 7.5 (Win ME)
Opera 4
Opera 3
Opera 2
Opera 1

Netscape
Netscape 8.1(Win XP)
Netscape 8.0.1 (Win XP)
Netscape 7.2 (Win 2000)
Netscape 7.1 (Win 98)
Netscape 4.8 (Win XP)
Netscape 4.76 (Win 98)
Netscape 4.5 (Win 98)
Netscape 4.05 (Win 95)
Netscape 3.01 gold (Win 95)
Netscape 2.02 (Win 95)

Safari
Safari 3.0 (v521)
Safari 2.0 (v418.9.1)
Safari 2.0 (v412)
Safari 2.0 (v401)
Safari 1.3 (v162)
Safari 1.3 (v158)
Safari 1.3 (v146)
Safari 1.2(v125.8; OSX 10.3.4)
Safari 1.1 (v100.1; OSX 10.3.2)
Safari 1.0 (v85)
Safari 1.0 (v74 beta 2)
Safari 1.0 (v73 beta 2)
Safari 1.0 (v64 beta)
Safari 1.0 (v51 update 1.0 beta)
Safari 1.0(v48 public beta)

AOL
AOL 9.0/MSIE 6.0 (Win XP)
AOL 8.0/MSIE 6.0 (Win XP)
AOL 7.0/MSIE 5.5 (Win 98)
AOL 6.0/MSIE 5.5 (Win 98)
AOL 5.0/MSIE 5.0 (Win 98)
AOL 5.0 (OS X)
AOL 5.0 (Mac PPC)
AOL 4.0/MSIE 3.02 (Win 95)
AOL 4.0/MSIE 4.01 (Win 95)
AOL 4.0 (Mac PPC)
AOL 4.0 (Mac 68k)
AOL 4.0/MSIE 3.0(Win 3.1)
AOL 3.0/MSIE 3.02 (Win 95)
AOL 3.0/MSIE 4.01 (Win 95)

Mozilla
Mozilla 1.7.8 (Linux)

Spiders/Bots
Googlebot 2.1 (New version)
Msnbot 1.0
Msnbot 1.0 Beta (v0.11)
Yahoo Slurp
Ask Jeeves/Teoma

RSS Readers
NewsGator 2.5 (Win XP)
Feed Demon 1.6 (Win XP)
Bloglines 2.1
SharpReader 0.9.6.0 (Win XP)

WAP Devices
Nokia 6630
Nokia 6100
BlackBerry 8703
iPhone (3.0 Mobile)

Toolbars
Google Toolbar 4.0
Hotbar 4.4.2
AskBar 3.0

Other
iTunes 4.2 (OS X 10.2)
iTunes 4.7 (OS X 10.2)
iTunes 7.1.1 (Mac PPC)
Zune 2.0
Facebook

Sweet! Take another browser off the browser compatibility testing list.

Read more here...

Want to download a few keepsakes or take a walk down memory lane? Check out Netscape's Product Archive FTP.

Back in 2006 I wrote the article The $60 Web Testing Toolbox with the intent of providing a cheap set of Web testing tools for testers. Most of the tools were mentioned because they provided me the most bang for the buck, but the reality is that good testing tools sometimes cost money. I wanted to share with you my complete Web Testing Toolbox minus the cheap factor. Developing software is not cheap so why does testing of it have to be? For the most part, these tools are inexpensive and have proven to provide great value. In my experience, each and every one provides enough value to justify the cost. Keep in mind I've pretty much lived in a .NET environment since 2001, so some of the things in my list tend to lean in that direction.

I'll keep this list updated as I see fit. I've provided a permanent link in the right navigation for permanent and quick access.

Also, If you're shopping for testing tools and you want an extensive list of what is out there, check out ApTest's tools section.

Microsoft is doing a Visual Studio 2008 Installfest at various locations across the country!

What specifically is "Installfest"? Tim at Microsoft says:

"Want the latest IDE toolset and start developing solutions in .NET 3.5 using Asp.net Ajax, Silverlight, LINQ, Astoria and all the good stuff? What are you waiting for?  There are various Visual Studio 2008 Installfests around the country.  My team in the west is no different, but maybe you've been having a hard time finding them?  Here's a list for all those happening in the west!  please feel free to blog and spread the word. We're providing a location, install dvds (these will be evals because the finals aren't pressed yet), and food/drinks.  This will be a good time to get some questions answered, get VS 2008 installed, play around, and hang out with your peers.  Each registered and attended individual will receive a special code to receive the fully-licensed Visual Studio 2008 Professional copy sent to them when the full package product is available to ship. Don't miss these events!!!  Registration is limited, so get in now and mark your calendars!"

If you're in Phoenix, our Installfest is shaping up to be December 20th. Watch the AZGroups calendar for details.

Understanding how to get at the controls in a modal dialog box when using SWEA is something I never had to worry much about in the past; the SWEA example code took care of all of those that I had previously encountered. Recently I needed to automate a "Choose file' dialog, and since there was no example, I had to buckle down to figure out how to get at the controls in that dialog. To understand the controls of this particular modal dialog, and also wanting to make sure I could handle all others I encountered in the future, I wrote some code that would dump all relevant things available in SWEA's DialogControl class when the DialogActivatedEventHandler was fired. By doing this, when a dialog was presented, I was able to scrape all the control information out of the modal dialog. The following code in my explorerManager_DialogActivated method did the scraping and dumping:

if (this.Debug == true)
{
    Console.WriteLine
        ("--Modal dialog info--------------------");
    Console.WriteLine
        ("Modal Dialog Title: " + dialogScene.Title);
    Console.WriteLine
        ("Modal Dialog ClassName: " + dialogScene.ClassName);
    Console.WriteLine
        ("Modal Dialog DialogHWND: " + dialogScene.DialogHWND);
    Console.WriteLine ("Modal Dialog XML: " + dialogScene.Xml.ToString());
    Console.WriteLine
        ("----------------Controls in Modal dialog-----------------");
    foreach (DialogControl dlgCtrl in dialogScene.Controls)
    {
        Console.WriteLine("Control name: '"
            + dlgCtrl.Name + "' is a type of: " + dlgCtrl.GetType().ToString());
    }
    Console.WriteLine("----------------------------");
}

//Manage the 'Choose file' dialog box
if (dialogScene.Title == "Choose file")
{
    if (dialogScene.ControlExists("Edit_0"))
    {
        dialogScene.DialogEdit("Edit_0").Text = 
            this.ChooseFileLocation;
    }
    if (dialogScene.ControlExists("Open"))
    {
        dialogScene.DialogButton("Open").Click();
    }

Very often I see automators using the sleep method in their Web browser automation scripts. This makes me cringe and sigh (a heavy sigh that is). Improper use of the sleep method is simply a bad idea.

For example, take the following pseudo-code:

link.click( )
sleep(30 seconds) #waiting for flaky frame to load
verify.html("My text in frame")

Siiiigh... This pattern plain sucks. For two reasons in particular:

1. If the page loads in 1 second, you just wasted 29 seconds.
2. If the page loads in 31 seconds, your script just broke, or the test failed.

From my personal experience, people typically write this evil code because:

1. The way the automation library ties into the browsers "the page has completed loading" events are unreliable so they are left with sleep to ensure the page is really there before testing continues.
2. They're in a hurry, debugging takes too long, and providing an average sleep fixes the issue (temporarily) 

I'd like to help you out with #1; I'll leave #2 up to you.

When detection of completion events isn't reliable within a framework, you need to come up with your own type of detection. What you're left with when the "the page has completed loading" event has come and gone is verification of object(s) on the page to see if the page is there. However you can't just sleep and then verify, instead of sleep I recommend a power nap instead. A power nap method could like this: 

powerNap(time sleepTime, HTMLobject objectToWaitForName)
{
    NapTimeStart = Time.now

    while not (HTMLobject.exists)
    (
        TimeGoneBy = Time.now - NapTimeStart

        if (TimeGoneBy >= sleepTime )
        (
            break
         )
         else
        (
            sleep(1 second)
         )
    )
}

With this loop you will only waste a maximum of 1 second once your HTML object is actually found on the page. Your automation code/test could then look something like this:

link.click( )
powerNap(30 seconds, lastObjectThatLoadsOnThePage )
verify.html("blahblah")

Now this method isn't guaranteed to work, however your success % is going to go up dramatically. The caveat to this approach is the fact that just because the HTML object exists doesn't mean the entire page is there. If you are attempting to do something with that object and it relies on other objects that aren't loaded yet, then you might get an error (e.g. click a link that has a reference to a method in a .js include file that isn't fully loaded yet). To avoid this you'll want to pick an object that you know loads last for the page and sometimes that can be tricky. You have to play with it a bit, but if you put the time in now, you'll get rid of those wasteful and sometimes test breaking sleeps.

As an added bonus you could take the example method provided and go one step further to wait for multiple HTML objects to be present before moving on. With a method like that you could verify the existence of several HTML objects on the page without using assert or verify methods (much like SWExplorerAutomation's Scene object).

In summary, without looping use the sleep method for debugging and nothing more, instead of sleep take a powerNap.

I've spouted off before why I think being a tester-developer is important. Do you want to work your way into being a .NET Tester-Developer? Microsoft is trying to make it easy as pie for us. With videos targeted at the beginner with ZERO experience this is a perfect opportunity for you to dive in and start down the path of Tester-Developer.

Are the videos too slow for you speedy QA mind? With Windows Media Player 11, right click the play button and select "Fast Playback", set back and soak them in at 1.5x speed.

I found a great article over at the Google Testing Blog entitled Automating tests vs. test-automation. The article does a great job of summing up some things about automation that I had to learn myself over the years. The last set of bullets is oh-so-true, and if you are starting down the automation project path be sure to heed the advice:

To summarize, I figured out that a successful automation project needs:

The ACE TEAM over at Microsoft has released a tool that does static analysis for cross site scripting (XSS). Confused about what a static analysis tool is? The ACE TEAM explains it well:

There are two types of web application vulnerability scanners: dynamic and static. Dynamic analysis tools are also called penetration testing tools. You point such a tool at a live application; the tool begins crawling the web pages in the application and throws test strings at each of them. The effectiveness of a penetration testing tool is therefore dependent on its ability to go through all the use cases in the application. Most tools in the market, if not all, are not very good at it. Static analysis tools on the other hand scan the application source code or binaries to detect programming errors. Consequently, they offer 100% coverage and are able to identify many more vulnerabilities than penetration testing tools. XSSDetect is a static analysis tool.

Get all the specific details on how XSSDectect works here. XSSDetect is now in Beta and is a free download here. Question is, who is going to run the tool in your company? QA or Dev? We all know who should be running it first. Don't we? DON'T WE? Okay then, download it, play with it, get to know it, present it to the Dev team (making you look REALLY smart and up to par on the latest technology), and then talk them into running it against their code before release for testing.

Life is good when your app isn't vulnerable to XSS. Life is better when the developer finds the XSS vulnerability before you do.

Tuesday evening I had the opportunity to attend my first Phoenix Software Quality Association meeting (PSQA) where another Rock-Star GoDaddy employee Lauren Snyder presented an "Introduction To Watir". It was really great to see her enthusiasm for automation and on top of that she is a great presenter. The presentation provides a HEAP of information about Watir and would be valuable to anyone diving into Watir.

Heh, get it? DIVING INTO WATIR.

Awe, never mind.

Lauren gave me permission to post the presentation here for your consumption:

Introduction To Watir.htm
Introduction To Watir.ppt (Powerpoint 2003, 8 MB)

If you're in or are interested in software quality assurance, and are in the Phoenix area, be sure to check out the PSQA site and come to our next meeting.

We testers have a wind-up.

No, not  wind-up as in wind us up and watch us go like a little Chihuahua nipping at the heels of software...

I'm talkin' rear back on the haunches, LOAD UP, and DELIVER:

Except in my software testing mind I see it more like this:

My wind-up consists of:

• Gathering requirements
• Writing test cases 
• Getting to know the application
• Setting up the test or automation framework

Once I wind up, the blow is...

crushing

devastating

ruinous

tragic

That poor, poor, POOR piece of software. Let the defects flow like the river Nile.

But really, is it any one persons fault?

Marketing defined it as (notice its a little dark and hard to see):

Developers did their best with the shady requirements and time constraint:

And well, testers tend to deliver a smack-down and exposed the frightening truth:

Wind-up and deliver that blow Mike Erbes!

I've been following the Microsoft hacking blog %41%43%45%20%54%65%61%6d (can you decipher that?) for a while now, but its content isn't necessarily what you may be thinking... posts on how to hack Microsoft products? Nope, instead the content of the blog is written by hackers that now work at Microsoft. You know, the white hat kind. If you can't beat 'em, hire 'em? There aren't a lot of posts, but when there do post they tend to be interesting.

The latest post First Line of Defense for Web Applications - Part 1 had a lot in common with a recent project I'd been working on, and the image they created to illustrate their point is a beautiful summary for their post and also for what I was working on. What the image portrays is a very common problem, a problem that a lot of testers don't know how to test for, or help enforce with good requirements:

This series should be good and I look forward to the next post. So should you.

While I'm at it, make sure and check out a little less of a white hat perspective at http://ha.ckers.org/. There is a ton of valuable stuff here, stuff that has worked its way into my test cases.

Back in October of 2006 I wrote an article for Better Software Magazine entitled The $60 Web-Testing Toolbox. Wrong or right, I'm posting it here (3.9 MB) since I have a heck of a time tracking down where I put the digital copy whenever the subject comes up. The $60 toolbox is not my ideal toolbox, but it proves the point that you can get a great set of tools on a low budget. I've got about half an article written about my ideal Web testing toolbox (see my placeholder on the right entitled "Brent's Test Toolbox). Hopefully someday soon day that "Coming soon!" link will become active.

In my ideal software development world... "Quality is everybody's job". If you could help negotiate and create that world where quality is everybody's job, what would your Quality Contract Proposal to the project team be? Recently I submitted to my project team something similar to this:

Team Quality Tasks

• Communication
  • Daily, morning 5-10 minute stand-up meetings for quick progress and issue reporting
  • Team notification of document and code changes
• Documentation
  • Requirements required
  • All documents are stored in a common area for easy access, check-in/out and versioning (e.g. SharePoint)
• Code Reviews
  • Large code reviews are scheduled during stand-ups
  • Peer code review before check-in for small or quick changes
• Use of Development & QA metric tools

Developer Quality Tasks

• Unit Testing
  • Web pages containing business logic will be unit tested with NUnitAsp
  • Other layers with VSTS Unit testing (Microsoft.VisualStudio.TestTools.UnitTesting) or NUnit 
• Builds
  • Dedicated build box for build integration.
  • Build and test on check-in.
  • Build and unit test run schedules (e.g. hourly, daily).
  • Automated build success and failure notifications.
  • Build Integration rules
    • Get latest from source control, merge, compile, and unit tests success before source control check-in.
  • Official builds released to test
    • Starts on functional section code completion. 
    • Developer collaboration and agreement on release time and contents.
    • An owner for release to test (owner checks with team to ensure release readiness).
• Provide "Quality Hooks" that don't impact performance or usability (e.g.)
  • Use of special image naming.
  • Use of special ID and name attributes.

Tester Quality Tasks

• Participation in design meetings and discussions.
• Review and feedback of requirements draft document.
• Review of Developer unit tests to provide suggestions for improvement as well as get a better feel how/where to provide further test coverage.
• Testing
  • Functional
    • Testing starts on functional section code completion
    • Utilizes test case libraries for common tests (HTML elements)
  • Automation
    • Scripting starts on functional section code completion
  • Performance, Load, & Scalability (typical and per functional unit if needed)
  • Other

Some of these items may apply to your company or team and some may not. Each has its benefit, each has its cost. In my experience the benefit outweighs the cost. Wouldn't you agree that this Quality Contract Proposal is a good place to start? Do you think it it may be a little far fetched for your company or current process? How about starting smaller then? Throw the list out there and get the team to commit to 25% of the items. When the project is over, take the lessons learned and infrastructure built from that 25% and use them in the next project, but now since that 25% is tried and true, ask for 25% more. Then 25% more...Get my drift?

Baby steps are okay. Be a salesman for software quality, negotiate that Quality Contract!

Roy Osherove is in the process of writing a book entitled The Art of Unit Testing. You can read the first chapter for free and/or buy a copy in advance and read up to chapter 6. I took the time to read chapter 1 last night and I like what I'm seeing so far. This book will definitely be added to my collection. What I like the most about this book hit me on the first page:

"Ever since that first project that failed, I’ve been compiling best practices for unit tests and using them on the next project. Every time I get involved in a new project, I find a few more best practices."

Roy is giving us honest to goodness best practices from real world experiences; both success and failure with unit testing. If you're a developer, architect, development manager, tester, tester-developer or managing quality, give the practices in this book a try and if it works for you and your team...spread the word.

Bret Pettichord, the lead developer for Watir has posted his original and recently rewritten Watir tutorial class workbook and all course materials. If you're new to Watir and want to learn it, this is a great start parting point. Go check it out it's free.

Here are my pre-requisites for agile defect reporting:

1. You need to have GREAT communication with the developers. What is great? A mutual respect for each other. An ability to talk at the level that both tester and developer can understand.
2. You need to be able to quickly show your defect, especially if it is complex and hard to explain. What is quick? Email is NOT quick. I think quick is:
   1. Recreating in person.
   2. Using the phone.
   3. Sharing your desktop (Windows desktop sharing feature).
   4. Instant Messaging screen-shots.
3. YOU must be able to conduct regression on the defect fix. Nobody knows the defect better than you. Especially since we're going to be light-weight on the documentation (see The 10 minute rule below)

Here are my guidelines for agile defect reporting:

1. The 10 minute rule: If the developer can't devote time to you within 10 minutes then ask "When do you think you'll have time to take a look at this with me?". If within 30 minutes then press the 30 minute snooze button, make a note of it, and continue testing. It's important to make a note of the defect at this point. A simple one liner will suffice, just enough info to kindle your memory of how you found it or how to recreate it. It is likely that you'll have a few more defects to share by the time that 30 minutes is up, so your notes will help keep you on track when it's time to share them. I use Microsoft OneNote (they have a free trial).
2. The 30 minute deadline: If you can't share the defect within 30 minutes then put it in the defect tracking tool.

Experience has proven that there is one downside to being defect agile (sharing within the 30 minute window): there is little defect history if the defect wasn't entered into a defect tracking tool. So you are limited to your notes. I think in the last 2.5 years this bit me twice when a defect reoccurred and we were looking for how we fixed it (which would only work if your developers are detailing their fixes in the defect report).

Being bitten only twice is worth the price if you ask me. Hundreds of defects reported in an agile like fashion can save countless hours,days, or weeks for a tester and developer over the course of a project.

Jason Haley over at JasonHaley.com does a utility review every month and this month he reviews the SQL Server query performance testing tool.  I haven't tried it, but Jason says it works well, and at first glance the interface looks very easy to use. You can download the tool at DataManipulation.com.

Why should you become a Tester-Developer?

1. The software industry needs them bad. The demand is high and there are few.
2. The high demand is an indication that the industry is moving away from the waterfall mentality to a more agile environment where testers are involved earlier and need to understand and write code for testing.
3. You will likely make better money because you will be a better tester.
4. It will be easier to get jobs with testing AND developing skills.

I hear the question quite often: What is a tester-developer? Steve Rowe sums this up quite well:

"Test Developers are the heart of a modern test team.  There was a day when you could get away with hiring a few people to just use the product and call that a test team.  This is no longer the case.  Products are becoming more complex.  The lifespan of products is increasing.  More products are being created for developers instead of end users.  These have no UI to interact with so simple exploratory testing is insufficient.  To test complex products, especially over an extended lifespan, the only viable solution is test automation.  When the product is an API instead of a user interface, testing it requires programming."

What do you have to lose? Pick your poison, C#, VB.Net, Java, JavaScript, TSQL, Python, Ruby, or whatever is applicable to your company or to your interest. Find a few books, use tutorials on the Web, and most of all just start writing code. Keep the future in mind; what do you think you're going to gain my learning Cobol? Do you think that will be applicable in the next job you take? Be patient and don't rush, learning to program isn't a fast or easy process.

This message was brought to you by the Push to Become a Tester-Developer Foundation  (PBT-DF) where we've been pushing testers to become tester-developers since...well, since you started reading this blog post. Be all you can be join the PBT-DF.

I recently had to do a little research on the options available for automating Safari and FireFox on Mac OS X. Below are the possibilities I came up with. I haven't tried any of the options out but first impressions make me lean towards FireWatir and SafariWatir. Here are my findings with a few very raw thoughts and comments:

Watir (FireWatir & SafariWatir)

Good ol' Watir. Seems to be quenching the thirst on the Mac too!

Selenium

Large list of browser/OS support. Downfall is that it runs through a proxy or from inside a frame in the browser. How do you know that frame isn't conflicting with your web app? What if you were running for days looking for memory leaks? If the automation tool is in a frame in your browser and the browser explodes, how do you know if it's the Selenium or the browser? As noted here, Selenium sucks because: hacky workarounds for popups, SSL, & XSS, is not native browser driver, infrequent releases, Selenium-RC Safari "really sucks".

WebDriver

Eggplant  

Appears to simply drive OSX windows/functions. Doesn't seem to know of or care about the DOM or the content in the browser. A Code example in the forum give me the impression that you can open Safari and use HTML controls using a "HotSpot" . Slight script interaction with Safari here too (open browser enter URL). Validation appears to be solely based image recognition.

iMacros for Firefox

Quick and dirty. It would be possible to automate something in minutes.

CoScripter for FireFox

Works in FireFox as a plugin. Appears very well thought out and very well done. Uses a wiki to view test details and edit test scripts. Pretty cool! Demo here. Made by IBM.

Lately I've been doing Web service testing using Visual Studio 2005 Team Edition for Software Testers. When I started testing using Team Edition, the features available in the "Software Testers" version were new to me and very confusing. I was able to quickly overcome my inexperience with a set of videos found at ASP.net. There are 27 in all, and the series is entitled: Videos on how to improve the quality and integrity of your software solutions.  I didn't watch every one of them but I intend to. Check 'em out if you're just getting started with VS 2005 for Software Testers and I guarantee it'll give you a serious boost and make you productive a lot faster. 

There is also a great video series available on MSDN entitled How Do I?.  They have a specific subset based on testing entitled: How Do I: Use .NET Code to Customize Coded Web Tests?

Avoid the fumbling around and watch somebody do it first. It makes things so much easier!

Mladen Prajdic over at SQLTeam.com has created a UI to compliment SQL Server 2005's TableDiff.exe.

Download it here.

I haven't talked about HTTPWatch since version 4.1 but last week I had a chance to work with the  5.0 version and I have to say once again: I LOVE IT.  These guys keep me hooked with the simplicity, new features, and ease use with their application. What I'm loving the most in 5.0 is the quick visual of traffic activity (blocked, DNS Lookup, connect, send, wait, receive, TTFB, & network):

The second thing that caught my testing eye was the grouping by page feature which allowed me to expand/collapse by pages:

HTTPWatch 5.0 continues to impress. It's still by far my preferred tool for viewing HTTP traffic while testing Web applications. You just can't beat the usability, quick availability from within the Internet Explorer window. And to boot, it has an interface for use with Web application automation! Give it a try for free here.