All About Caching and How Visual Studio Simulates Browser Caches

  • Article

Simulation of Browser Caching during load tests

In a VS load test that contains Web tests, the load test attempts to simulate the caching behavior of the browser.  Here are some notes on how that is done:

  • There is a property named on each request in a Web test named "Cache Control" in the Web test editor (and named "Cache" on the WebTestRequest object in the API used by coded Web tests).
  • When the Cache Control property on a request in the Web test is false, the request is always issued.
  • When the Cache Control property is true, the VS load test runtime code attempts to emulate the Internet Explorer caching behavior (with the "Automatically" setting).This includes reading and following the HTTP cache control directives.
  • The Cache Control property is automatically set to true for all dependent requests (typically for images, style sheets, etc embedded on the page).
  • In a load test, the browser caching behavior is simulated separately for each user running in the load test.
  • When a virtual user in a load test completes a Web test and a new Web test session is started to keep the user load at the same level, sometimes the load test starts simulates a "new user" with a clean cache, and sometimes the load test simulates a return user that has items cached from a previous session.    This is determined by the "Percentage of New Users" property on the Scenario in the load test.    The default for "Percentage of New Users" is 0.

Important Note: When running a Web test by itself (outside of the load test), the Cache Control property is automatically set to false for all dependent requests so they are always fetched; this is so that they can be displayed in the browser pane of the Web test results viewer without broken images.

Comparing new users to return users

There is a property in the Load Test Scenario settings for "Percentage of new users". This setting has impact on a few different aspects of the load test execution. The percentage is a measure of how many of the simulated users are pretending to be "brand new" to the site, and how many are pretending to be "users who have been to the site before".

A better term to describe a new user is "One Time User". This is because a new user goes away at the end of its iteration. It does not "replace" a different user in the pool. Therefore, the term "New User" should be considered to be a "One Time" user.

The "Percentage of New Users" affects the following whether the tests contained within the load test are Web tests or unit tests:

  • The value of the LoadTestUserId in the LoadTestUserContext object.     This only matters for unit tests and coded Web tests that use this property in their code.     On the other hand if you set the number of test iterations equal to the user load, then you should get a different LoadTestUserId regardless of the setting of "Percentage of New Users".
  • If you are using the load test feature that allows you to define an "Initial Test" and/or a "Terminate Test" for a virtual user, then it affects when the InitializeTest and TerminateTest are run: for "new users" (a more accurate name might be "one time users", the InitializeTest is run for the virtual user, the "Body Test" is run just once, and then the "Terminate Test" is run.     For users who are NOT "new users", the InitializeTest is run once, the Body Test is run many times (until the load test completes), and then the TerminateTest runs (which might be during the cool-down period).

The "Percentage of New Users" affects the following Web test features that are not applicable for unit tests:

  • The simulation of browser caching. The option affects how the VUser virtual browser cache is maintained between iterations of Tests. "New users" have an empty cache (not the responses are not actually cached, only the urls are tracked), "return users" have a cache. So if this value is 100% all Vusers starting a Test will be starting with an empty browser cache. If this value is 0% all VUsers will maintain the state of the browser cache between iterations of Web Tests. This setting affects the amount of content that is downloaded. If an object sits in a Vuser cache and if the object has not been modified since the last time the Vuser downloaded it, the object will not be downloaded. Therefore, new users will download more content versus returning users with items it their browser cache.
  • The handling of cookie for a Web test virtual user: new users always start running a Web test with all cookies cleared.   When a user who is not a "new user" runs a Web test after the first one run, the cookies set during previous Web tests for that virtual user are present.

 

The below graphs (taken from test runs in VS 2010) demonstrate the difference between a new user and a return user. The graphs are based on a 10 user / 50 iteration run, but with different percentages for "new users" on each run.

 

How cache settings can affect your testing and app performance

This article shows how changing the caching settings in your Visual Studio tests and on your web server can impact your test. It also shows a real world demonstration of the difference between NEW and RETURN users.

 

Notes:

  • All 4 tests above were run for the same duration with the same number of users executing the same test.
  • Although the numbers do not match exactly, they are close enough to show the behavior of the tests. The discrepancy is due to a few things, including cool down of the test and the possible misalignment of the query I used to gather data from the IIS logs.
  • The IIS Log items for "200 –OK" and "304-Not Modified" were gathered using LogParser and the following query:

SELECT

    sc-status, COUNT(*) AS Total

FROM *.log

WHERE

    to_timestamp(date, time) between

        timestamp('2010-02-12 02:13:22', 'yyyy-MM-dd hh:mm:ss')

    and

        timestamp('2010-02-12 02:18:22', 'yyyy-MM-dd hh:mm:ss')

GROUP BY

    sc-status

Comments

  • Anonymous
    December 11, 2013
    All About Caching and How Visual Studio Simulates Browser Caches - Testing The Testing Strategy - Site Home - MSDN BlogsContreraznt&automated