Performing code coverage
I was trying to find way out to perform code coverage for my product. VSTS is the immediate answer. Write unit tests, select binaries which require code coverage results and simply run tests. Code coverage results window has it all at the end of the test run. But product’s code base is 2 years old and has got no automated unit tests. Writing automated unit tests for the entire product before the planned milestone date is highly not possible. Definitely post to our release date, we will have a plan and schedule automating unit tests.
For now, I went through few approaches and came across the tool called ‘Vanguard’. To my understanding, vanguard is formerly called as DesktopCC – an internal code coverage tool used by Visual studio team system. In short, this tool has a Control Window. Load binaries which needs code coverage to be performed and start monitoring. Provide .coverage output file where coverage results must be saved to. Exercise activity / tests to consume these binaries. Once all done, stop monitor. Use Coverage View window and open the .coverage file and view the code coverage results. Under hood, following happens (which I became aware from Joc’s blog https://blogs.msdn.com/ms_joc/articles/406608.aspx ): In the VS installation directory - Microsoft Visual Studio 8\Team Tools\Performance Tools\
Step 1: vsinstr -coverage myassembly.exe
Step 2: start vsperfmon –coverage –output:mytestrun.coverage
Step 3: Execute unit tests
Step 4: vsperfcmd –shutdown
The resulting <mytestrun.coverage> file containing code coverage results can be opened and analyzed in Visual studio.
Note on Step 3: Since I don’t have automated unit test cases, I can perform an exercise of executing list of unit test case scenarios manually when the binaries are being monitored. Automation is required here to make this manual process repetitive and speedy. If its a vc++ project, ensure /Profile compiler switch is provided to build the project before performing step 1.
I was looking for tools which has advanced capabilities and recently came across one such tool called 'Magellan' through my team mate, Himanshu.
A quick look inside Magellan: The Magellan Code Coverage Framework, developed by the Test Effectiveness group in the Programmer Productivity Research Center (PPRC) – From the house of Microsoft Research. BBCover, one of the tools in the Magellan framework, inserts instructions into a compiled program to collect information during a test run. The tool gathers information about the basic blocks and arcs of a binary. A basic block is a set of instructions that is always executed together; and an arc is any point in a function where the execution jumps from one block to another. The CoverMon application saves the test results to disk, and then imports the information into a SQLServer database. This helps a developer or tester to get immediate information for a new or changed component. Another tool, Blender, integrates test data from multiple versions of a binary in the Magellan database. Because of Blender, large amounts of coverage information from one version can now be applied reliably to new versions.
Integration capabilities with Sql Server database, Product Studio and Source depot:
The Magellan user interface is called Sleuth. It provides over 50 views into the database. Testers can see detailed reports for the binaries, source files, classes, and functions in the tested application. When Sleuth is integrated with Source Depot, it helps testers to see their test coverage line by line in the source code. Integration with Product Studio allows testers and developers to share comments about the data. The Magellan Object Model is a COM interface that test teams can use to share Magellan data on the Web.
I haven't got an opportunity to use this tool yet! But, I will get to know more details on this tool from Himanshu as he is currently working on it. Will blog sometime later when I get more details.
Comments
Anonymous
April 21, 2008
PingBack from http://microsoftnews.askpcdoc.com/?p=3443Anonymous
September 29, 2008
Hi, I am experiencing an issue when using NUnit with Vanguard, is Vanguard compatible with NUnit? Thanks!