Rethinking Testing – Dev Test Challenges

In my last post, I had mentioned that I would pick a specific area of testers challenges and share with you how we are addressing it. In this part of the series, I want to address heads on a very key issue – respect for testers.

There are testers from all walks of life and background. Some very technical, some experts on the product they are testing, some are specialists and often they are also developers. While testing itself is recognized as one of the most critical components of application delivery, it’s worth a wonder whether the great majority of testers do get the respect they deserve. Why is it an accepted fact of life that testing gets crunched when a project is running over time? What would really happen if a testing organization said “We cannot sign off on this release” and put their foot down? If these situations intrigue you, I invite you to wonder with me, why they haven’t changed in the history of software development.

One key aspect of quality assurance (as the word is described often) is the assurance that the software is of a certain quality. Do all testing teams out there feel like they in-fact have an impact on Software Quality and can assure it, or do they just focus on finding bugs? In an interesting informal survey at a very well represented testing conference, a lot of testers wondered if they added value to their organization. I invite you to wonder with me why this could be the case.

Testers work hard, they are smart, and often are the closest a project team can get to the end user before actually releasing their product. But let’s examine the following issue closely.

Tester files a bug

Developer reviews it – marks it as “Need More Info”

Tester collects a lot of data and adds to bug

Developer marks it as “No Repro”

Tester has spent 20 hrs on this bug already, and developer has marked the bug no-repro (after having spent some chunk of his/her time).

Each time this happens, the developer is frustrated that the testers are not doing enough and the tester is frustrated because he/she is always on the defensive – it’s like the bug is not a bug unless a developer finds it (innocent until proven guilty). How frustrating is that for a tester who just spent a good portion of his week on this bug?

To this date, what has happened is that testers have created a world in which developers have little to no space. They don’t use the same tools the developers use, they don’t see the bug in same way a developer does, and they are not empowered enough to share this data across the teams. Why is this hard to change? The fact is, it is not. Except that, for the longest time we’ve built teams and tools that separate developers and testers. At Microsoft, we are excited to bring these two teams closer together and deliver better interaction. We want to break the silos between dev and test teams and build bridges.

In this series of posts, I want to give you specific details on HOW we are doing this. To best describe this, I want to segment the testers in the world into three broad buckets. This will allow me to not only break up the blogs, but also introduce to you the various solutions that we are bringing to market in the near future to address each of the areas.

1. Engineering Support / Developers: These typically are developers who write code for pure customized automation testing or hard core libraries to be used by others. They would likely stay on a development career and do unit testing and deep white box testing

2. Specialist Testers: These typically are people with engineering backgrounds, are formally trained in QA and continue to write code/scripting for automation, performance, security and other advanced testing.

3. Generalist Testers: These typically are testers that DO NOT have a computers/engineering back ground, have at some point used the application that is under test and are motivated to make sure the users of the software get the best product they can deliver. They typically DO NOT aspire to be developers or technical coders. They are very savvy when it comes to testing, and deeply understand end users of the application.

Our research shows that 70% of testing done in the Industry is manual testing done by generalist testers, with the other two categories making up the remaining 30%.

While Microsoft is a fairly new and unknown player in the test market, we have shipped our Unit testing, and Load and Performance testing solutions in 2005 as Visual Studio Team System Test Edition (VSTS) and then continued investing in the VSTS 2008 release. These testing solutions are successfully being used by many and we continue to get great feedback from our customers.

For Visual Studio Team System 2010 release, Quality is one of the two pillars for the release (driving Business and IT alignment being the other). We have announced a wave of products that make up the test offering that solve critical customer pain points. The following figure shows how our offerings map across the development and testing teams and especially maps to the different competencies across testers in an organization.

Caption: A single snapshot of the Microsoft Test Offerings in Visual Studio 2010

Our test offering is built on top of Team Foundation Server that serves as a single repository allowing different team members access to information they seek via tools, programmatic access, reporting, and other means. With virtualization playing a very key role in development and test environments, Lab Management capabilities are built in a fully integrated way as a core ALM capability that allows developers to focus on coding and testers to focus on quality while eliminating bug ping pong and tedium of setting up complex environments.

While it is a commonly available toolset, our customers are asking us for integrated solutions that allows test case management to co-exist with all other work items and project assets that govern the success of a project. We have built our test case management framework grounds up to allow different team members a complete view of test cases and all related artifacts.

Microsoft Test Runner allows generalist testers to execute their manual test workload very easily via “Capture & Replay” technology and empowers them to log bugs that are rich in information. Rich bugs allow developers to quickly recreate bugs and avoid the dreaded bug ping pong that is no one’s favorite.

“Coded UI” testing allows you to bring quality earlier into the cycle by making additional tests as part of your build verification tests.

This hopefully gives you a good overview of what we are doing. In future posts I will go into more details of each of the areas. I do hope you will come back to read more specifics about each of these areas of testing.

Here’s wishing you, and your families and friend, a Very Happy and Prosperous 2009!

Comments

• Anonymous
  December 31, 2008
  PingBack from http://www.codedstyle.com/rethinking-testing-%e2%80%93-dev-test-challenges/
• Anonymous
  January 05, 2009
  GertD on Creating T-SQL Static Code Analysis Rule Amit Chatterjee on Rethinking Testing - Dev Test Challenges...
• Anonymous
  February 08, 2009
  [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez
• Anonymous
  February 13, 2009
  The comment has been removed
• Anonymous
  April 20, 2009
  This post finishes up the two part topic on Diagnostic Data Adapters in VS 2010, and how they help developers