Make Your Performance Work Count: The 20% Guideline

I recently read Steve Seow's book, Designing and Engineering Time: The Psychology of Time Perception in Software. Steve is an expert on performance engineering at Microsoft, and really understands the human element and psychology behind software performance.

A really interesting take-away is that users do not notice changes in performance below a certain threshold. Research has shown that for durations under 30 seconds, users do not detect differences up to 18%. Steve's rule of thumb is that users really won't notice performance changes of +/- 20%. So if you are looking at setting goals for a performance improvement, or considering performance against a competitive product, your goals should be to gain at least 20% over the baseline measurement.

Likewise if you are considering whether or not to fix a performance regression, 20% is a good measurement to consider. If the regression is less than 20%, users likely won't detect the degradation, so you may be better off spending your effort elsewhere.

Of course this is a slippery slope. If each release you allow a 20% degradation, users certainly will notice and stop using your software. So it is important to maintain and meet performance goals that will keep your users happy with your software. One of my performance testing customers ends his emails with this quote:

"If it is fast and ugly, they will use it and curse you; if it is slow, they will not use it."

-David Cheriton in _The Art of Computer Systems Performance Analysis_

Ed.

Comments

• Anonymous
  January 02, 2009
  PingBack from http://www.codedstyle.com/make-your-performance-work-count-the-20-rule/

• Anonymous
  January 02, 2009
  Funny. Unfortunately Microsoft is using this rule again again so from W95/W98/W2K/WXP/WV now we have 1.21.21.2*1.2 = 2.0 slowdown, don't we ? He ? Me noticed ! (Not speaking my hardware is from that time sooo faster).

• Anonymous
  January 02, 2009
  Hah, that is funny! I almost put a caveat in the post about accumulating 20% degradations, of course it adds up to greater than 20%. And no, this is not a wide-spread, in use rule at MS. :) BTW, you'll like Win7, it has a bunch of perf and resource usage improvements over vista. Ed.

• Anonymous
  January 04, 2009
  Ed Glass from the MS Load Test Team wrote a post: Make Your Performance Work Count: The 20% Rule Basically

• Anonymous
  January 07, 2009
  The comment has been removed

• Anonymous
  January 08, 2009
  Performance & the 20% Rule I was reading the post from Ed Glas , Make Your Performance Work Count:

• Anonymous
  January 09, 2009
  Very interesting! Thanks for the tip. By the way, I have been trying Windows 7, and it does seem fast. I was not impressed by startup/shutdown, but sleep/resume was like lightning! Really looking forward to it.

• Anonymous
  January 10, 2009
  clflare, The rule strictly applies to human perception, so it only applies to performance measurements that the user experiences directly. When I considered some typical response times this made sense, +10% would be hard to notice, I think I'd start noticing at +20%: time +10%   +20% 2     2.2    2.4 5     5.5    6 10    11     12 30    33     36 The specific case you pointed out was 30s. Really 30s is unacceptably long for any response time. I stop paying attention and start on something else at about 10 to 15s. This is also backed up by Steve Seow's research, I have another post coming on that topic. That said, I do think going from 30 to 36 would be noticable, less than that probably not. A good example of a performance measure this applies to is the time it takes a web page to load. What really counts there is the first point at which the user can interact with the page and read it, it doesn't necessarily have to be completely loaded. A case you point out and I agree with that it does not apply to at all is anytime the measurement may something like a function time, say String.Concat that the user doesn't experience directly. If String.Concat took a 20% hit it would have a far greater performance consequence than 20% for some operations since it will be called many many times. The other interesting case you brought up is something like typing. Steve Seow categorizes these operations as "instantaneous", and there is interesting research on this as well. The threshold that humans perceive an operation as instantaneous is 100 to 200 ms. If you go over 200ms, it is no longer instantaneous. So I agree with you on that as well. Ed.