Using Code coverage in Black Box testing ain’t no rocket science :) Overcome your CODEPHOBIA

I deserve few more comments here as I travelled 6 km and 30 mins extra to write this blog as in the excitement of writing this i forgot to get down at my stop and ended up at a place where the driver had to tell me that it was the last stop :P

“Yeah our developers are not doing code coverage and hence our code looks ugly” OR “we are not doing white box testing so we really cant measure code coverage"   Heard it before ? Now Rocky (our tester) was in one such meeting recently and he was the only odd man out who believed otherwise. It took us some time to realize that it was the “CODEPHOBIA of the testers which is the major resistance when it comes to measuring code coverage and not as much as the actual process”.  

This reminds me of one of my favorite commercial which has this one liner “Jhooth bolte hai woh log joh kehte hai unhe darr nahi lagta, darr sabko lagta hai. darr se aage bado, kyunki darr ke aage jeet hai” (For people who don’t understand Hindi: “People who say they aren’t afraid of anything are plain lying. Everybody gets scared (and its very human) but the key is to face it and then only you emerge as a winner”

When i interviewed the teams, I came to this understanding that the testers feel when they are no experts in .Net, Java, Ajax etc then how can they measure and improve the code coverage.  I don’t completely agree by this notion as measuring code coverage doesn’t necessarily need very strong programming knowledge. Provided that you have a basic understanding of any programming language like C, Perl etc and a strong willingness to learn, you can still ramp up pretty fast to the extent which is required for you to be able to measure/improve code coverage. Understanding code is much easier than writing it :) and all it needs is aptitude and basic understanding of the syntax and semantic of the language in question. I am saying this based on my own experiences where I went an learned the technology when it was required to be able to accomplish targets like measuring and improving code coverage.

Now if you are saying “Why we are even talking about code coverage”? 

Requirement coverage is ensured by tracing the requirements to your test cases and ensuring user needs are met but what about the traceability between the code and the test. In other words, what happens if there is code which is not exercised  by your test and results into unexpected behavior at customer’s end.

In short because Test Coverage = Requirement Coverage + Code Coverage

Lets accept it that “requirements” and “code” are two different entities and validating just one of these would be unfair to the other one.

Now if agree that code coverage can be done by black box testers and there are no major obstacles then lets move on to “How” part. Well, so if there is a will then there is a way and fortunately in this case there are many ways. We need to choose what fits best for us.

Step 1: Measure you baseline and set target: is to see where you stand now by measuring the BASELINE code coverage for your project (as you cant improve what you cant measure) and then decide your realistic TARGET code coverage number. Now don’t get me wrong for highlighting keyword  “realistic”. Yes, I am no conservative and everybody want 100 out of 100 (no less) but the practitioners would tell you its same as “exhaustive testing” which we know doesn’t come for free. So you got to decide on a number by seeing the criticality of your application and a trade off between the money spent in increasing code coverage vs. risk of missing a defect due to insufficient code coverage.

for e.g. if your test (manual + automated) provides you 40 % coverage today then it becomes your baseline and you can set a target say 70-80 % for the future releases by doing all calculation on (investment vs. ROI)

Step 2: Improve and measure (continuous)

Then you can go and measure release-over-release IMPROVED code coverage and then compare it against the baseline to detect the trend (if there are any improvement), till you reach your target. Once you achieve your target code coverage number, you can raise the bars to take it to next level and continuously improve it.

 

There are tools that can be used to start these trace using command line options which will instrument and track the code execution (statement coverage, path coverage, branch coverage etc) based on the tests being run (both automated & manual) and then help you generate reports to see the impact/coverage of your tests.  I have given few links below which will point to some of these tools and the tool selection/adoption is at the reader’s discretion.

Is there any process which can be used to achieve this. Answer is YES. Read on.

 Code Coverage process:

I have come up with a generic iterative code coverage process which we can be used for measuring & improving code coverage while test execution (black box or white box).

1. Start the Trace using Code Coverage tool

    Here you install, configure and start the CC tool before starting your test execution.

2. Test Execution – Phase 1

    You are running you test cases or scenarios or conditions  (manual or automated whatever)

3. Measure Coverage & Identify areas where coverage is low

At this point lets say when you have done one pass of testing or have run your test cases once, you can Generate code coverage reports using the CC tool in question and IDENTIFY rooms for improvement where code coverage is not good.

4. Add Test cases / conditions to improve code coverage

  Now you know which segment of the code was not covered by our test so you can go and either

  a) write new test cases or scenario to cover those parts

  b) Include the test data which can exercise those missing conditions

5. Test Execution – Phase 2

  You start again and this time you run these new test cases / conditions with new test data.

6. Measure Coverage again

You generate code coverage reports AGAIN and you see if there are improvements

Loop: Keep repeating steps 2 to 6, till you reach your target. End the trace

Pointers to few CC tools:

https://www.codecoveragetools.com/index.php/coverage-process/code-coverage-tools-c.html

https://www.codecoveragetools.com/index.php/coverage-process/code-coverage-tools-java.html

 

If you still with me then you are among those who are willing to learn new things in life and i wish you a ''Happy Code Coverage”. Go celebrate it. If it helps then why not use it. Technical knowledge shouldn’t be an excuse for not doing it. \

After all it is no rocket science.

PS: CODEPHOBIA in my dictionary is “fear of code; not only writing even reading or trying to understand it”

Comments

• Anonymous
  November 24, 2011
  The comment has been removed
• Anonymous
  January 11, 2012
  The comment has been removed