Running Tests with Code Coverage

  • [アーティクル]

As I talk to customers I find that there are many different work flow models. I use the term “work flow” to describe all the things you do to get complete your objectives. That work flow can be documented and shared with others at the office. Having a positive and efficient work flow is important, otherwise we get cranky.

Meeting the needs of the different work flows can be difficult. In this post, I’ve outlined three common work flow situations with steps to successfully collect code coverage data. These scenarios were built around different models of access to source code because that seems to be the underlying issue customers experience.

Every scenario minimally requires product binaries, product PDBs, test binaries, and Visual Studio Team System. For better, richer, and more flexible experiences you’ll need more, like access to test and/or product source code. In the end, you just want code coverage data so it doesn’t matter how you go about it. Pick a method that works for your setup and enjoy what VS Team System has to offer. J

For more information about how you can go “off-roading” with code coverage, I’d recommend looking at John Cunningham’s blog.

Scenario 1: Local Product and Test Sources

This is the most ideal scenario. One has the product and test projects loaded in VS. One has full ability to build both which provides the most flexible experience.

This is probably a developer or a tester who work directly on the product. They are trusted to and it is logistically possible to have full access.

Gathering Steps

  1. Load the product and test sources in a single solution in Visual Studio Team System 2005.

  2. Perform a full build.

  3. Open the Run Configuration you intend to use for this run (‘localtestrun.tesrunconfig’ is the default)

  4. Select the Code coverage page in the resulting Dialog

  5. Check the Binaries you wish to be instrumented

  6. Execute the tests you desire to run, either through Test Manager or Test View.

Review Code Coverage data

  1. When tests are finished, open the Code Coverage window and review the results.

Scenario 2: Product Binaries and Test Sources

This is probably the most likely scenario. One has product binaries (and PDB files!) from a daily build. One has access to the test sources and can build test binaries. This nearly provides the same flexibility as Scenario 1.

In this model, the user may be the same developer or tester as in Scenario 1, but they may be encouraged to only test against “official” builds from a common source. This could also be a person testing offsite from core development who receives regular drops of builds from the development team. They may be interested in gauging the comprehensiveness of their testing as measured by code coverage, or this activity may have been requested by the development team. In the latter case, the code coverage results file will be sent to the development team for analysis.

Gathering Steps

  1. Load the test sources in Visual Studio Team System 2005.

  2. Copy the desired product binaries and PDB files locally.

  3. Build the test sources.

  4. Open the Run Configuration you intend to use for this run (‘localtestrun.tesrunconfig’ is the default)

  5. Select the Code coverage page in the resulting Dialog

  6. Check the Binaries you wish to be instrumented

  7. Execute the tests you desire to run, either through Test Manager or Test View.

Review Code Coverage data

  1. When tests are finished, open the Code Coverage window and review the results. If you don’t have access the source code, you will only be able to see the blocks covered etc., rather than full detail and coloring of the source code.

Scenario 3: Product and Test Binaries Only

This scenario requires the most work and provides the least amount of benefit, but unfortunately it is all too common. All the great work VS does for you when you have your own test project is gone because you only have test binaries now. Additionally, this could include running manual tests so there might be no automation to run at all.

That said the scenario still works. You can get code coverage data. Also, if that data is shared with someone who has access to source code, then they will get full benefit from the results as they can see code coverage detail and line coloring.

Gathering Steps

  1. Copy the desired product binaries, product PDBs, and test binaries files locally.

  2. Instrument the product binaries:

    1. Open up a Visual Studio 2005 Command Prompt (Start | All Programs | Microsoft Visual Studio 2005 | Visual Studio tools | Visual Studio Command Prompt).

    2. Change to the directory where your binaries are located

    3. Execute “vsinstr –coverage <assemblyname>” for all your binaries. e.g. If your binary name is ‘coolproduct.dll’, then type “vsinstr –coverage coolproduct.dll”.

  3.  Start the Code Coverage monitoring process

    1. Return to the previous command prompt

    2. Type “vsperfcmd –start:coverage –output:outputFilename.coverage”.

  4. Execute your tests

    1. For automated tests, navigate to the directory where your test binaries are located and execute “mstest /testcontainer:TestDll.dll”.

    2. Run manual tests, if you have them

  5. End the Code Coverage monitoring process

    1. In the command prompt opened earlier, type “vsperfcmd –shutdown”. This shuts down the coverage monitor and saves the coverage data to disk.

Note: There are alternatives to manually executing these commands, but they are slightly more complex.

  1. Start VS and create a test project

  2. Add your production DLLs as references to the project

  3. Open the Run Configuration you intend to use for this run (‘localtestrun.tesrunconfig’ is the default)

  4. Select the Code coverage page in the resulting Dialog

  5. Check the Binaries you wish to be instrumented

  6. Save the runconfig file

  7. Exit the IDE

  8. Copy the Product DLLs and Test DLLs into to the correct place along with the runconfig

  9. Execute “mstest /testcontainer:testdll.dll /runconfig:runconfig.testrunconfig”

This will instrument and execute the tests, while correctly starting & shutting down the coverage collector. If you still wish to instrument the binaries yourself, then you still need a dummy-dll to be instrumented. The tools do not start the coverage monitor unless the tool itself has instrumented a DLL.

Review Code Coverage data

  1. Open the .coverage file in Visual Studio.

Comments

  • Anonymous
    June 15, 2005
    VSTS Quality Tools - Running Tests with Code Coverage
    David Williamson and Dominic Hopton do a great...

  • Anonymous
    July 03, 2005
    VSTS Quality Tools - Running Tests with Code Coverage
    David Williamson and Dominic Hopton do a great...

  • Anonymous
    July 04, 2005
    The comment has been removed

  • Anonymous
    May 07, 2007
    Three common work flow situations with steps to successfully collect code coverage data. These scenarios

  • Anonymous
    June 19, 2008
    I am trying to get code covergate for my WCF services. My methods in the services will call another dll to complete the functionality. How can I get the code coverage for that dll? I have tried to add that dll in the test project, but not able to get the code coverage. Thanks, Kavitha

  • Anonymous
    July 07, 2008
    As part of my weekly duties since joining the Quality Tools team, I have been providing code coverage