How Does the User Pacing model differ from the Number of Tests Started model

  • [アーティクル]

In my post on User Load Profiles, I talked about choosing one of two different load models, based on a couple of the options in Visual Studio. However, I have still received a lot of requests to clarify these two models and to show how I can apply one of the models to the testing I am going to do. Hopefully this post will help clarify some of this.

Each model can easily be selected regardless of what webtests you have created and both can easily be ramped up to different load levels. The primary difference between the two is how Visual Studio handles the think time between each iteration of the virtual users:

  • “Based on Number Of Tests Started” executes the test and then waits for the amount of time provided in the “Think Time Between Test Iterations” property for the scenario. The start time for the first iteration for each vUser is the start time of the vUser.
  • “Based on User Pace” gets a rough estimate of how long each iteration can execute and still leave enough time for all subsequent iterations to complete. (e.g. 30 tests/user/hour = one test per user every 2 minutes.) The actual start time for the test iteration will be random and can happen anywhere within the calculated window.

Based on User Pace is much more predictable regardless of response times and Based on Number Of Tests Started is much better at driving load as fast as the system can respond. The images below show the user responses for two virtual users in each of four different load tests:

  • 100 users – Based on Number Of Tests Started
  • 100 users – Based on User Pace
  • 1000 users – Based on Number Of Tests Started
  • 1000 users – Based on User Pace

The four graphs below are representative of the User Activity graphs in Visual Studio, but simplified to make the explanation easier. The blue boxes represent the execution of a single iteration of a test by a single user. The space in between blue boxes is the Think Time Between Test Iterations. The gray line at the bottom represents the number of minutes into the overall load test execution.

In tests 1 and 2, we see that the response times are the same, but that test 1 completed more iterations (10) than test 2 (5).

numTestsLightLoad

userPaceLightLoad

In tests 3 and 4, the load against the system has caused the response times to slow down. Test 3 (which is identical to test 1 except for the number of virtual users) now only completes six iterations per user. (NOTE: The iteration in yellow is a test that started but did not complete in the allotted time and therefore isnot counted in the results. See this post for details) Test 4 (which is identical to test 2) still completes 5 iterations per user.

numTestsHeavyLoad

userPaceHeavyLoad

The best way to think about this is to observe the “white” space in the images above. This represents the “think time between iterations. ”

  • With both load profile models, the System Under Test (SUT) controls the time it takes to complete an iteration.
  • With Based on Number Of Tests Started you control the amount of think time (or pacing).
  • With Based on User Pace, Visual Studio controls this, and it uses the SUT’s response time to set it.
  • If you want to drive load against the system quickly and irrespective of the throughput, then use Based on Number Of Tests Started.
  • If you want to drive load against the system and know that you are providing a known throughput, then use Based on User Pace.