Test planning using Camano

The concept of a test plan

Just like architects plan and document the design and requirements of software products, and just like developers plan and document how they will be implementing it, testers also plan and document their strategy to verify that such products meet those specifications and requirements. This documented strategy created by testers is called a test plan. Great test plans are those that are able to effectively verify the quality of a product (given the requirements and specifications) through the use of effective manual and/or automated test cases.

Test plans in Camano

Camano is the internal codename for the test case management and test runner solution that we have been working on as part of our next major release (Visual Studio Team System 2010 Test Edition).

Test plans in Camano fit the definition that we already talked about, by providing testers the tools they need to document their strategy. But it also go beyond that - letting them organize, execute and track the progress of the different artifacts on their plans (test cases, configurations, runs, etc). This is how a test plan looks like:

I will throw in some questions for you along the way. Are these test plan properties (name, description, iteration, etc) shown here enough? Leave a comment if you think there are key attributes missing?

When and why does it make sense to use a test plan?

A test plan makes sense whenever there is an application, specific feature, or specific goal (such as testing localization) within a context of an application. Test plans in Camano will help test managers track testing from its initial design to its completion, and will help testers track their test cases, manual and/or automated runs, analysis, and defects.

Test design: adding/organizing test cases to/on a test plan

The testing design starts whenever testers start thinking and documenting test cases that will help them uncover defects throughout the whole lifecycle of a product. Camano enables this design process in a very simple way.

A tester can add existing test cases (maybe there are test cases that can be reused from other projects or features) or just create new ones on any test plan. These test cases can also be grouped in what we call static or query-based suites. This is what an organized test plan with test cases could look like:

As you can see in the left pane, suites are used to organize tests (which appear in the right pane), and progress on the test plan can be seen right there as well. Does this seem easy and straightforward to use?

Test cases are ready to be run either manually or in an automated fashion right away, and the progress of plans can be tracked in a straightforward way as well. A usually complex task turned into a simple and fun effort!

Test organization
Test organizations can be as small as one tester doing the whole job, and up to tens or hundreds of people working on a project. Roles go from test managers, to test leads, and individual testers. Camano makes it easy for the whole team to work together, and at the same time enables each person on the team to focus on stuff that he/she needs to get done.

Test managers or leads will be interested in knowing how the testing effort is going. They might want to know, for example, how many test cases exist for a given feature, and how many have been completed so far. They might also want to know the owner for each of these test cases. They can easily view all this information from any given test plan:

Testers may want to track planned test cases, and may want to filter based on those that are assigned to them. They may also want to specify different configurations for them. Camano will let them do all these and more…

How to know when testing is done?
Saying that we’re done testing is never an easy task. In order to have the right answer, the correct test design will need to be figured out, and the actual implementation of that test design will need to be completed. Camano will not test for you, but it will give you the right and simple-to-use tools to accomplish this goal: Easy test design, easy tracking of the effort, and easy reporting on the status of test cases; all these things right out of the test plan. If the right design is in place, the answer will be a straightforward one: “We’re done testing”.

What else is there in Camano?
This post has focused on test planning; but there is much more in Camano. As soon as test planning is done, execution is what makes a test plan look good. Camano makes running tests a simple and efficient task. Afterwards, there is a whole integrated story that includes analyzing runs, linking and tracking bugs, viewing reports, looking at recommended tests to run on new builds, running tests remotely and collecting data such as code coverage, and even creating test environments that testers can share with their developers. Are we missing any features that you as a tester think we have to ship with this version?

When we go out to the field and demo the product to customers, we’ve heard that they love the whole integrated story that lets testers easily test a product and create bugs against it; also the fact that developers get all the information they need in order to be able to repro the bugs quickly and fix them accordingly.

Stay tuned for further blog posts describing all these features in Camano that integrate the tester and developer stories together!

Can’t wait for future blog posts?

Then go ahead and try the latest bits! You can get them here:
https://connect.microsoft.com/VisualStudio/content/content.aspx?ContentID=9790

Please let us know if you love our product, and if you think we’re missing anything.

Comments

• Anonymous
  February 16, 2009
  PingBack from http://www.clickandsolve.com/?p=9358

• Anonymous
  February 16, 2009
  is this the new layout of camano? are their any large changes...

• Anonymous
  February 17, 2009
  Responding to Clemens - yes, this the current and new layout of camano. We have made some big simplications of the model. As the tool is still under development, we may do some more changes to the user experience.

• Anonymous
  February 19, 2009
  I really like the integrated concepts I saw so far. Just one thing should be taken care of: Getting test reports easily out of TFS to be printed and signed off.Using VSTS2008 I am currently struggling with two problems:Putting (unit-) test results into a word document.Managing manual test cases and reporting the results. For the first I use a xslt file for transforming the trx file of a test run and then copy the resulting html table into a word document. Not nice, but working.For the latter, I use a test case work item instead of manual tests, because I can much better put this information into an excel spreadsheet, which provides for a good tool for documentation for other stakeholders who cannot access TFS. And, that spreadsheet can be put into a document management system, be printed and approved by signature. This is especially neccessary for regulated projects, e.g. medical device development, and companies relying on paper based documentation (which are many of the really big ones I know.)Quite useless for my report are the links between specifications and related test cases, because this info is not available in Excel. Additionally, I would like to report at least the IDs of the linked bug work items of failed test cases.For the time being we put some effort into manually transferring the IDs of the links into text fields, so that we have that information available in the spreadsheet.Eventually, I would like to have a spreadsheet containing the following columns: test case ID, referenced spec ID(s), test case title, test case description, tester, test result, referenced bug ID(s), ...As said initially, I like the upcoming stuff and can't wait to have them available. I even like the current possibilities when being online. The only really frustrating thing is to get meaningful reports out of TFS easily. There is always a lot of manual work just for the reporting.Regards,Erik Steiner

• Anonymous
  February 20, 2009
  Erik - I am glad you like the integrated concepts.Reporting will be much better in VS2010. I will add a post on reports to this series and highlight the capabilities there.

• Anonymous
  March 02, 2009
  For Test Planning, I would like to see Requirement (ex: actual associated requirement/s for this test case, PreCondition (ex: user must have admin or manager rights) and  Expected Results (ex: user should be able to add/update/delete a record, etc.) added.  I would like to be able to export it to a Word doc. I assume you can enter a defect that will be associated to this test case and the code files used to correct the problem discovered.

• Anonymous
  May 07, 2009
  The comment has been removed

• Anonymous
  May 19, 2009
  I am very excited proud and proud to introduce the new testing features you’ll find in dev10. In addition

• Anonymous
  May 26, 2009
  I spent a week in early May at the Software Design Review (SDR) / Knowledge Transfer at our Microsoft

• Anonymous
  September 02, 2009
  Hi Amit,My question is regarding access to the Test Plan by other stakeholders of the project that might or might not have access to VSTS 2010. Is there a way to make them public ? Is there a way to publish them in Sharepoint ?Thank you and congratulations for the product.

• Anonymous
  February 11, 2010
  I'm very interested in these Test Case Management features -- but can't tell if they've been released?  Is this version available for purchase? How do I get them?!?