How to get a free lunch in testing

How to get a free lunch in testing.

I hear people talk about getting testing "for free" quite a bit. The reasoning goes something like this. “If we do a complete functional test of the product, we don’t need to test underlying APIs, stored procedures and other low level code. We get coverage there for free.”

This line of thinking is deeply flawed. It’s a sure way to increase your sustained engineering costs and put off finding bugs until later. Those bugs just get more and more costly the longer they linger in the code.

The real way to get a free lunch in testing is to start at the bottom. Test early and test often. If you have a large code base that has never gotten any low level testing it can seem expensive to go test it. You are busy, you have scenarios to run. So it seems like you could get the underlying code tested “for free.” That's a trap. Think hard before you let that code into the wild.

Your free lunch costs more than you think

Testing the user scenarios and considering the underlying APIs, stored procedures, tables, etc. tested just postpones finding the bugs. It also impacts the underlying quality of your product. It seems to work on the surface, but there are scary bugs waiting to get you. The trap is that you feel good about the product. It passes all the scenarios you can think of. Your automation runs and reports 100% pass. You release the code and customers violate your expectations (they always will) and the bugs come pooring in.

This is bad for several reasons. First, we don’t want the customers to find bugs. Realistically they will find some, but we want to minimize this as much as possible. Second, the bugs they find have a chance to be monsters. The fixes for these bugs are either expensive or hacky. Third, the longer a bug goes undetected the more expensive it is to fix. These bugs are fully amortized and cost us the most.

Take the long term view. Make sure your foundations are solid.

You can get a free lunch in software. You just can’t have it today. Concentrate testing on the foundations of your software. 100% code coverage can tell you for sure is that you ran the code the program didn’t crash. Making sure the API’s are robust and can stand up to a full combination of boundary tests and fuzzing means you are much less likely to have to patch a major problem after you ship.

The free lunch you get comes when customers report bugs. The class of bugs they report will be niggling, irritating and mostly skin deep. Best of all they are easy to fix. Your sustained engineering effort will be small, and your valuable engineering team can focus on V-Next of your product.

Engineering good solutions is like investing wisely.

It takes vision, discipline and patience to see your investment pay off. Remember there is no such thing as a free lunch. But, if you invest wisely you can use the profits to buy yourself a lunch or two when your investment matures.

Think hard about the approach you take when you test your products. Customer focused teams will find a way to invest in the core quality of the products. This is how we can delight our customers.