Challenges in Test – Knowing Your Product

  • Article

So you accepted the challenge of being a tester, got your machine setup with the latest version of the code, installed your product, and your manager has given you a hot new feature to test.  It seems so easy. You think you can just focus on that feature, but you better think again.

Problem introduction

Take for example a feature called Query in Forefront Identity Manager (FIM).  You start out on the feature & learn that it takes a query in the form of XPath & then returns the results of that query.  You have some standard set of operators that you expect to find such as =, >, <, and or, not along with some functions like starts-with, ends-with, dateTime, etc.  The specific operators are not the important component here, but if you are so inclined you can view the details here.

Knowing your product

As a tester you might be prone to take this problem & run with it to create a bunch of test cases around the various combinations of operators & data values.  Initially you find several bugs & you are keeping your developer pretty busy.  You even think about some of the various buckets of tests like boundary, localization, globalization, performance, stress, security, etc.  Are you now ready to declare your testing complete?

One of the things a good tester needs to do is fully understand the problem space they are working in & also how it may be changing over the course of the product lifecycle.  For example in this case the design of FIM leverages the XPath filters of Query in a feature called Sets.  These Sets are then used in another feature called Management Policy Rules (MPRs). MPRs are core to the product in determining both if you are allowed to do something.  These MPRs also can trigger another feature Workflows which can then perform various actions in the product.

Yikes! You have just gone from a tester of a single feature to something that is deeply involved in 4 different features which are ultimately core to your product.  And note that these features are likely covered by different developers, testers, & PMs.  So you may not be responsible for these other feature areas, but you will need to know & understand how your feature interacts with them.  Even better if you are in active development the design may be rapidly changing as your team reacts to requests in future milestones, so what you knew yesterday may not be true today.

What’s the Impact?

With knowing your product you can now start to evaluate the level of coverage you have & where you might want to add more coverage.  One example of how you will need to use this knowledge is justifying the impact of a bug.  Let’s say you found a bug in Query which when searching for “Smith” exclusively returns all people with the last name of Smith, but also Smithers are returned at the end.  It is the day before you ship the product & the question will invariably be asked: “Can this bug wait until the next patch, service pack, or release?”  Or put more simply, “What bad things will happen if we don’t fix this right now?”

Now your knowledge of the product is key. When the feature was originally designed it was just used for searching a list of users or groups, so returning a few extra users would be no big deal right?  I mean who would page through the list of all Smiths in a phone book to see that there are a few extra results at the end?  But wait remember from above what you learned about the product & understand where else your feature gets used which is granting rights in the system & kicking off workflows.  So now this bug has increased in importance since you would potentially be granting rights to users & starting processes which are not intended.

So as you can see as a tester you better know your product almost as good as anyone else to help ensure you have the appropriate level of coverage & you don’t get the luxury of just knowing a single feature area but need both depth & breath across features.  If you want to really know your product then no better way than to test it.

Comments

  • Anonymous
    March 08, 2011
    Again, excellent insight - it really forces you to think about the permutations of tests along with all the various interdependencies. How do you coordinate with the other SDET's that are testing dependent features?

  • Anonymous
    March 08, 2011
    Yep, coordination of coverage is a problem to be aware of.  I may do a topic on it, but basically as with many things it comes down to coordination. Some might do this informally & others do this more formally with some form of document.  But you have to come to some conclusion & explaination between teamspeople of who is covering what & how you plan to overlap.  Otherwise you could at worst wind up with no one covering it or possibly duplicating work & wasting time that could be applied to getting some other cool feature going.