Profiling Tip: Amdahl's Law and Long Sampling Runs

  • Article

A lot of my graduate work was in the area of parallel computing so we spent a lot of time with Amdahl's law.  Amdahl's law states that if you can speed up some fraction F of the program by some speedup S, the overall speedup of the program is:

1 / [ ( 1 - F ) + ( F / S ) ]

So if you can speed up half the program by 8x, F = 0.5 and S = 8, the overall speed up is only about 1.8x.

By the same token, if you profile a program and find that you can really speed up a part of your program that is only 5% of inclusive time, the best possible speedup is 1.05x.  The moral of the story of course is that you should focus your effort on optimizations that will affect the greatest possible portion of the program.

Which leads me to the question of really long sampling runs.  One thing we've noticed about our users is that they tend to run their app for a really long time under the sampler.  I'm not sure I understand this technique.  You can get a statistically significant number of samples pretty quickly.  Remember that you can use vsperfcmd with the -timer option to set the sampling frequency but even with the default on most processors you are going to get several thousand samples even in a few seconds.  That should be more than enough to tell you where the hot section of your program is.  It doesn't really matter if you get enough samples to tell if some part of your code is 0.5% of your inclusive time or if it's actually 0.7% because Amdahl says you should be ignoring it anyways.  Likewise, you won't care if another function is 50% or 53% of samples because you know that's the bit that needs to be fixed.

I asked RicoM about this a few days ago and he told me that he brings people in to the MSFT developers labs, warms their apps up and then just takes about 15 seconds worth of samples and its usually enough to find where all the allocations / time is coming from.  One possible reason that he suggested where this might be a useful scenario is when you are trying to figure out why an app runs well in the first hour but then slows down in the second hour, so you might want a longer run to try to catch whatever changes.

But it's certainly possible that I'm missing something...  Does anyone have a good reason to do longer runs for everyday sampling jobs?

Comments

  • Anonymous
    May 24, 2005
    In the games industry, we usually have several different states that our app can be in, and each one needs to be profiled.

    So generally we have a script that we follow that runs the application through each state (launch, menu, in-game, etc.) and we profile each state and the transitions between states individually.

    And yes, we even profile launch. Given the amount of assets the average title needs to load and potentially convert at launch, knowing when conversion starts taking more time than file I/O is a useful piece of information.
  • Anonymous
    May 25, 2005
    Visual Studio Team System

    Yesterday marked the one-year anniversary of the public announcement of...
  • Anonymous
    May 25, 2005
    hey Michael,

    I wouldn't count that as a long profiling run. You are just doing several different short profiling runs, no? I totally understand why you would need to capture each phase of the program, but I'm not sure why people sample the same phase for a really long time. Thanks for commenting though. It's always interesting to hear how different the way various people use perf tools is. What profiler do you use?
  • Anonymous
    May 31, 2005
    We usually end up using profilers built into the engines we work on. When you're profiling not only native code, but interpreted script, p-code script, JIT'ted script, etc., all running at the same time, we usually need the specialty profilers to properly identify where the load is coming from.

    One other reason that we'll build the profiler into the product is that when we're working on memory-constrained systems (like the Xbox or PS2), quite often the overhead from a full profiler will actually bump our product up over the memory threshhold on the test kits.

    To see how profiling is often done in game development, take a stroll over to Millenium-D and Millenium-E and check out John Olsen's articles in "Game Programming Gems."
  • Anonymous
    June 09, 2005
    I’ve pulled together all of the technical articles and walkthroughs from the various team member blogs...
  • Anonymous
    June 09, 2009
    PingBack from http://insomniacuresite.info/story.php?id=8269