Dogfooding Visual Studio Test and Lab Management Products ….

Dogfooding is a rich tradition at Microsoft. It essentially stands for “eating your own dogfood”. That is, product development teams are expected to use the product that they are building in their day to day work life, to ensure that the product is really ready for paying customers. You can find more about this practice from this blog post from several years back.

In my posts you have read about my excitement about the Test and Lab Management capabilities that are part of the VS 2010 wave of products. I am very excited about the many innovations that we are building for our customers (who are the testers and developers and business analysts in product development teams), and I love talking about them.

In keeping with the practice of Dogfooding, it then was very important for me to make sure my own development team members were using these products on a daily basis, and ensure that the products lived up to the high standards of usability, reliability, and capabilities that I know my own team needs from tools, for building world class products.

This dogfooding effort has been an important focus for my team for the last several months. All of the testers in the team have moved onto using the products on a daily basis. Their motto has been to “Be the first customer of the Test and Lab Management features, and accomplish all VS 2010 QA tasks using VS 2010 features” . The result have been great!

It is time now to share this dogfooding story with you. I hope this also inspires the readers who are part of development teams themselves, to go-live with these products. As you know, Visual Studio 2010 is currently in a go-live Beta state – and you can get more details about it here. We are also real close to releasing our next public go-live build (which is going to be an RC (Release Candidate) build.

So, let me get started on the story …

The first important point to make is that the Visual Studio Test and Lab Management product development teams are a highly distributed team. We are located in four locations across the globe, as noted below. In our teams, we have about 40 team members focused on testing the products that we are building as part of VS 2010.

For our dogfooding we use a distributed lab that uses our Lab Management product, built on top of Team Foundation Server (TFS). We have a TFS server in Redmond (known as the Pioneer server), and we have a distributed lab topology built on top of it. Here is how the lab topology looks like

The above diagram needs a bit of explanation. On the left hand side of the diagram you notice the TFS machine – it shows the application tier (which is a network load balanced setup) and the data tier machines. This is an “enterprise” scale setup of TFS – and you can get more details of this configuration from Grant Holliday’s blog post. Next notice, the “SCVMM” (System Center Virtual Machine Manager) system on the top right – this gets licensed as part of the Lab Management product, and essentially manages the host machines where virtual machines will be installed, and the library of virtual machines that a development team will use. The SCVMM server for our lab is also located in Redmond. Finally you see that each of the sites in my development team (for example, Hyderabad or Raleigh) have their own sets of machine which serve as the VM hosts in their local location, and also a library machine in the local location that stores the VMs and templates that are used in the site. This topology works great for my distributed team – all of the data intensive operations of creating and cloning virtual machines are done in the site where the VMs are being used, whereas the central TFS server and the SCVMM server orchestrates and coordinates the lab across the sites.

I should also point out that the above TFS server serves as the single repository for all of the development activities in our team – that is, it stores all of the source code for our products, all of the work items, test cases, test results, and lab environment assets.

As part of the dogfooding effort, my entire test team is now using the above Lab for all of their daily testing and reporting activities. For example, here is a view of the test plans that are owned by the team.

As my team has several feature crews who work on different parts of the product (like Load Testing, or Test Case Management Client or Server), you can see the plans created and used by the individual feature teams.

In the next diagram below, you can see more details of an individual plan – in this case an example from the Test Case Management client team.

You can see above how the test plan is composed of different test suites – for functional automation runs (FAR) or Performance runs or for exploratory testing where the suites are organized as a set of different “tours” through the product. The right hand side shows the test cases which make up the selected suite and various attributes about their current state.

Next is an example of a test plan run.

You can see the run details – the build that was used for the run and the results of the run. This gives you a sense of how the feature teams are organizing their testing, and going about the various types of test runs – executing and analyzing them.

The next important activity for them to generate repots so that we get an overall idea of how the testing is coming along.

My team heavily uses the excel-based reporting mechanism in TFS. I will show you a few examples, were specific reports can be generated by just a few clicks of the mouse.

As all of the test and development assets are already in TFS, the starting point for these reports is a query that we make on TFS. For example, below I show a query for test cases that are closed (that is, completed). You can then notice the option to create a report in excel.

This mechanism comes really handy, in creating a bunch of reports. Below are some examples of reports from in Excel.

The diagram below shows what percentage of test cases have been automated versus those that are manual tests.

The next report shows the breakdown of test work items by priority.

The final sample report shows the who are the top 10 owners of work items assigned to them.

The TFS server, as you know, also provides a SharePoint portal which in our case also works for an up to date dashboard of all of the QA activities in our team. Below is how this looks like – you can see how the latest information of nightly automation runs (NAR) and failure analysis readily available. The portal has been customized for a variety of other reports which show up as you scroll down the portal.

The entire development team is also benefitting immensely from all of the features that we have built for breaking down silos between developers and testers. As an example, take a look at an “actionable bug” or “rich bug” that has been filed by a team member.

Using our testing tool – the tester probably just typed in the title of the bug she found and hit enter. All of the rich and actionable information in the bug about the system information at the time of test, the exact steps that the manual tester executed, and various kinds of rich attachments are automatically captured in the bug. This makes it extremely easy for the developer to narrow down and fix the bug. In the above picture you can see the rich set of attachment that the developer has access to right from the bug report – for example, a video of the testing session and an IntelliTrace log for historical debugging. Moreover, in this particular case since the testing was also done on virtual machines using the Lab Management product, you can see we have a link to the virtual environment that the tests were run on. The developer can click on the link and get connected to the virtual environment right from their desktop!

I will close this post here – but there is more to the story. In my next post I will focus on the Lab Management part of the dogfooding, and how we are benefiting from the features in that product.

This is great stuff! All of the testing in my group is now being done using the products we are building, and it is great fun to experience first hand, the benefits that we are building for our customers. Quite often I will get a mail from a tester or a test lead about yet another thing they are proud of in the product – for example, here’s a quote from my Test Manager:

From: Dinesh Bhat
Sent: Thursday, January 07, 2010 10:59 AM
To: Dinesh Bhat's Direct Reports
Subject: Using Microsoft Test Manager in a real testing cycle

The product simply rocks in terms of ease of use and productivity! There are a ton of cool features. IMO, the product meets the bar for teams that use agile or formal process or for teams that are savvy about traceability. Moreover, you can really let the team work in a “friction free” manner. During this exercise, I did not pester the team a single time to know the status. In turn, I was fully aware of progress the team was making sitting right in my office! This is a great toolset for distributed development too …

The entire dogfooding effort gives me great confidence in saying that these tools are ready now for your use in your your development teams! We would love for you to adopt these tools now, and give feedback about your experiences. We just have a few more months before we close the development cycle on this release.

I know many of you are using these products already. I would love to hear about your experiences too – perhaps you can add those as comments to this post.