google v. microsoft, and the dev:test ratio debate

Every since I gave a talk at Google’s GTAC event here in Seattle this past October, I’ve had the chance to interact with a number of Google testers comparing and contrasting our two companies’ approach to testing. It’s been a good exchange.

Now it seems that, their toilets notwithstanding, Google focuses on testing with an intensity that is in the same general ballpark as ours. We both take the discipline and the people who do it seriously. But I think that there are some insights into the differences that are worth pondering.

Specifically, the disparity between our respective developer-to-tester ratios is worth a deeper look. At Microsoft the dev:test ratio varies somewhat from near 1:1 in some groups to double or triple that in others. At Google just the opposite seems to be the case with a single tester responsible for a larger number of bug-writing devs (clearly we have that in common).

So which is better? You tell me, but here are my thoughts (without admission of any guilt on Microsoft’s part or accusations against Google):

1. 1:1 is good. It shows the importance we place on the test profession and frees developers to think about development tasks and getting the in-the-small programming right. It maximizes the number of people on a project actively thinking about quality. It speeds feature development because much of the last minute perfecting of a program can be done by testers. And it emphasizes tester independence, minimizing the bias that keeps developers from effectively testing their own code.

2. 1:1 is bad. It’s an excuse for developers to drop all thoughts of quality because that is someone else’s job. Devs can just build the mainline functionality and leave the error checking and boring parts to the testers.

It’s interesting to note that Microsoft testers tend to be very savvy developers and are often just as capable of fixing bugs as they are of finding bugs. But when they do so, do devs really learn from their mistakes when they have someone else cleaning up after them? Are testers, when talented and plentiful, an excuse for devs to be lazy? That’s the other side of this debate:

3. Many:1 is good. When testers are scarce, it forces developers to take a more active role in quality and increases the testability and initial quality of the code they write. We can have fewer testers because we our need is less.

4. Many:1 is bad. It stretches testers too thin. Developers are creators by nature and you need a certain number of people to take the negative viewpoint or you’re going to miss things. Testing is simply too complicated for such a small number of testers. Developers approach testing with the wrong, creationist attitude and are doomed to be ineffective.

So where’s the sweet spot? Clearly there are application specific influences in that big server apps require more specialized and numerous testers. But is there some general way to get the mix of testers, developers, unit testing, automated testing and manual testing right? I think it is important that we start paying attention to how much work there really is in quality assurance and what roles are most impactful and where. Test managers should be trying to find that sweet spot.

Comments

• Anonymous
  December 09, 2008
  The comment has been removed

• Anonymous
  December 09, 2008
  Is finding the sweet spot of numbers really that important ? or should fostering a mindset that everyone is involved in quality more important ?( a cliche that is often used but how can it be put into practice ? )

• Anonymous
  December 11, 2008
  James, There is no universal answer to the question "where’s the sweet spot?" concerning dev:test ratios. The "sweet spot" is the one that works well for your individual company - be that Microsoft, Google, or Joe's Company. Clearly, the time and count of people required to test something is contextual - it depends on factors that may have little or nothing to do with how long it took to develop that feature.  It will vary by company and project, and may vary over time. Also consider:

• what counts as Development?
• what counts as Testing? see: http://www.sqablogs.com/jstrazzere/150/What+is+the+%22Correct%22+Ratio+of+Development+Time+to+Test+Time%3F.html

• Anonymous
  December 11, 2008
  Excellent points, all. I love the quote about innovation in testing. In my 'future' series I pointed out that quality will eventually be everyone's job. The point here is what is the proportion of quality-oriented people that takes us to that future faster?

• Anonymous
  December 15, 2008
  I don't think you can talk about dev:test ratio without talking about engineering process. For example, listen to the Channel 9 interview with Nachi Nagappan of the RISE group about his team's findings on TDD (test driven development). In his study, MS showed a 60-90% reduction in code defects at a cost of making the dev cycle 15-35% longer. Employing such a development process would likely change the ideal dev:test ratio. One thing I have little to no understanding of is how Test/QA is structured in other engineering disciplines (EE, Civil, etc). http://channel9.msdn.com/posts/Peli/Experimental-study-about-Test-Driven-Development/

• Anonymous
  December 16, 2008
  The comment has been removed

• Anonymous
  December 18, 2008
  The comment has been removed

• Anonymous
  December 23, 2008
  Testers can only test code.  How about design bugs in the design or architecture?  Who's responsible for vetting those artifacts?   One design / architecture error in a Microsoft product that replicates content from one farm to another assumes that if content is deleted and inserted again, and replication happens afterwards, the replication process fails because the newly inserted item already exists in the destination (it just tries to do an update or insert).

• Anonymous
  December 28, 2008
  The comment has been removed

• Anonymous
  December 29, 2008
  I'm not sure of the "optimal" ratio of Dev:QA, but I know what it is not.  Anything over 4 to 1 in my experience became unmanageable from the QA standpoint.  Meaningful testing never seemed to be done enough because of the turn around and release of the software.  QA was seen as "too slow" when in fact they were overwhelmed not slow. It made QA a true bottleneck on projects which also made the people in QA less happy. I've never worked in a 1 to 1 environment but it definitely sounds good.  I'm sure it gives the QA engineer the ability to accomplish the "should do"s and not just the "must do"s.

• Jim

• Anonymous
  January 01, 2009
  I'd love to know how closely the practices of the G v. MS testers align.  What do each spend the day doing - and at what abstraction level?  What time is spent on various tasks relative to each other?  What does the Venn diagram look like as to responsibilities? etc...

• Anonymous
  January 02, 2009
  You missed commenting on the 1:many environment - every development project goes through this phase, when substantially more people are using the software than ever worked to create it in the first place. In the 1:1 or Many:1 situations, one tester is testing the output, usually against a set of described use cases. Once the product reaches alpha, beta, or release stages, however, it gets tested by people who have no interest in the original designers' use-cases, only their own.

• Anonymous
  January 13, 2009
  I was triggered to read a blog post by James Whittaker, software architect at Microsoft, through an article

• Anonymous
  February 03, 2009
  [Nacsa Sándor, 2009. január 13. – február 3.]  A minőségbiztosítás kérdésköre szinte alig ismert

• Anonymous
  February 05, 2009
  [ Nacsa Sándor , 2009. február 6.] Ez a Team System változat a webalkalmazások és –szolgáltatások teszteléséhez

• Anonymous
  May 13, 2010
  Exactly why i like your blog. You tell iz like it is, no matter what. Whether it's Microsoft or Google product.

• Anonymous
  September 07, 2011
  The comment has been removed

• Anonymous
  May 11, 2012
  If you break ever story down and you have a task for each part of a story: design, develop, unit test, automated integration test, build, install, localize: Your team works on a story until it is done. The should take the task to write the automated integration test for each story and a team with a size of four to five people should only do one or two stories at a time, and each takes a different task. If the tester can keep up, you have enough testers. If he is good yet can't keep up, you need another one. Sometimes you might find you need one dedicated automated integration test developer and one developer who is a feature developer 50% and an automated integration test developer %50. In a four or five person team, you will see the ratio naturally arrive at 3 Devs, 1-2 Developers for Automated test.