Visual Studio Performance Testing Quick Reference Guide (Version 2.0) Published
Version 2 of the Quick Reference Guide has now been published. This guide is chock-full of information gathered through real experience with the product. This new version contains 97 new or updated articles.
Special thanks to Geoff Gray for once again spearheading this effort. Geoff is one of a handful of elite VS load test practitioners. As he encounter unexpected behavior or bugs in the product, or gets undocumented information on the internals or edge cases, he captures the info for the quick reference guide.
Here is a screen shot of some of the really nice artwork in the guide:
Here is a list of topics in the guide (the numbers after each article is the page number):
NOTE FROM THE AUTHOR
HOW IT WORKS
How Web Tests Handle HTTP Headers 9
General Info (including order of execution) of load and web test plugins and rules 9
Client Code does not execute because Web Tests Work at the HTTP Layer 12
File Downloads, Download Size and Storage of files during Web Tests 12
When is the “Run unit tests in application domain” needed? 12
How the “Test Iterations” Setting impacts the total number of tests executed 12
Test timeout setting for load test configuration does not affect web tests 13
How user pacing and “Think Time Between Test Iterations” work 13
Load test warmup and cool down behaviors 13
What is the difference between Unique, Sequential and Random Data Sources 14
Comparing new users to return users 14
Goal based user behavior after the test finishes the warmup period 17
Threading models in Unit tests under load 18
Simulation of Browser Caching during load tests 19
The difference between Load Test Errors and Error Details 20
How parameterization of HIDDEN Fields works in a webtest 21
Testing execution order in Unit Tests 23
How machines in the test rig communicate 25
Changing the Default Port for Agent-Controller Communication 26
How to Add Agents To A Test Rig 26
ITEMS NEW TO VS 2010
“Find” feature now available in Webtest playback UI 27
“Go To Web Test” feature now available in Webtest playback UI 28
Recorder Log Available 29
Add extraction rule directly from the playback UI 30
New “Reporting Name” property for web requests 31
LoadTestResultsTables now differentiate between GET and POST requests 32
Virtual user visualization now available 33
New Excel reporting features built into load test results 39
New Load Test and Load Test Rig Licensing and configurations 40
New test mix: “Sequential Test Mix” 44
Query String and FORM POST URLs get parameterized 46
New options on Load Test Scenarios 47
Loops and Conditionals 48
CONFIGURATIONS AND SETTINGS
How to Change the Location Where Agents Store Run Files 50
How to set a proxy server for web tests 50
How to configure Web Tests so Fiddler can capture playback info 50
Controlling the amount of memory that the SQL Server Results machine consumes 51
How to configure the timeouts for deployment of load tests to agents 51
How to set the number of Load Test Errors and Error Details saved 52
Multi-proc boxes used as agents should have .NET garbage collection set to server mode 53
Location of list of all agents available to a controller 53
NETWORKS, IP SWITCHING, TEST STARTUPS
IP Address Switching anatomy (how it works) 54
Gotcha: IP Address Switching is ONLY for WEB TESTS 54
Gotcha: IP Addresses used for switching are not permanent 54
How to Setup IP Switching 55
Troubleshooting invalid view state and failed event validation 58
Startup: Slowness Restarting a Test Rig with Agents Marked as “Offline” 58
Startup: Multiple Network Cards can cause tests in a rig to not start 59
Startup: Slow startup can be caused by _NT_SYMBOL_PATH environment variable 59
Startup: tests on a Rig with Agents on a Slow Link 60
“Not Bound” Exception when using IP Switching is not really an error 60
How to configure the timeout for deployment of load tests to agents 61
PERFORMANCE COUNTERS AND DATA
Customizing the Available Microsoft System Monitor counter sets 62
Performance Counter Considerations on Rigs with slow links 64
Increase the performance counter sampling interval for longer tests 65
Changing the default counters shown in the graphs during testing 65
Possible method for fixing “missing perfmon counters” issues 65
How and where Performance data gets collected 66
DATA AND RESULTS
Custom Data Binding in UNIT Tests 67
Verifying saved results when a test hangs in the “In Progress” state after the test has finished 67
The metrics during and after a test differ from the results seen. 68
How new users and return users affect caching numbers 69
data sources for data driven tests get read only once 70
Consider including Timing Details to collect percentile data 71
Consider enabling SQL Tracing through the Load Test instead of separately 72
How to collect SQL counters from a non-default SQL instance 72
How 90% and 95% response times are calculated 72
Transaction Avg. Response Time vs. Request Avg. Response Time 73
Considerations for the location of the Load Test Results Store 73
Set the recovery model for the database to simple 73
How to clean up results data from runs that did not complete 74
InstanceName field in results database are appended with (002), (003), etc. 74
Layout for VSTS Load Test Results Store 74
How to view Test Results from the GUI 75
SQL Server Reporting Services Reports available for download 75
How to move results data to another system 75
Load Test Results without SQL NOT stored 76
Web Test TRX file and the NAN (Not a Number) Page Time entry 77
Proper understanding of TRX files and Test Results directory 78
Understanding the Response Size reported in web test runs 79
ERRORS AND KNOWN ISSUES 80
CSV files created in VSTS or saved as Unicode will not work as data sources 80
Incorrect SQL field type can cause errors in web tests 80
Leading zeroes dropped from datasource values bound to a CSV file 80
Recorded Think Times and paused web test recordings 80
After opening a webtest with the VS XML Editor, it will not open in declarative mode. 81
Calls to HTTPS://Urs.Microsoft.Com show up in your script 81
Possible DESKTOP HEAP errors when driving command line unit tests 81
Goal based load tests in VSTS 2008 do not work after applying SP1 82
Using Named Transactions in a Goal-Based Load Profile can cause errors 82
Debugging Errors in Load Tests 83
Debugging OutOfMemory Exceptions in Load Tests 83
Memory leak on load test when using HTTPS 83
“Not Trusted” error when starting a load test 84
Detail Logging may cause “Out of disk space” error 85
Error details and stack traces no longer available in VSTS 2010 85
VSTS does not appear to be using more than one processor 85
Changes made to Web Test Plugins may not show up properly 85
Socket errors or “Service Unavailable” errors when running a load test 86
Error “Failed to load results from the load test results store” 87
Hidden Field extraction rules do not handle some fields 87
Test results iteration count may be higher than the max test iterations set 87
In flight test iterations may not get reported 88
Completion of Unit Test causes spawned CMD processes to terminate 88
Bug with LoadProfile.Copy() method when used in custom goal based load tests 89
Errors in dependent requests in a Load Test do not show up in the details test log 90
WCF service load test gets time-outs after 10 requests 92
Loadtestitemresults.dat size runs into GBs 92
TROUBLESHOOTING
How to enable logging for test recording 93
Diagnosing and fixing Web Test recorder bar issues 93
User Account requirements and how to troubleshoot authentication 94
How to enable Verbose Logging on an agent for troubleshooting 95
Error that Browser Extensions are disabled when recording a web test 95
Troubleshooting invalid view state and failed event validation 96
Troubleshooting the VSTS Load Testing IP Switching Feature 97
Troubleshooting Guide for Visual Studio Test Controller and Agent 99
HOW TO, GOTCHAS AND BEST PRACTICES
How to call one coded web test from another 111
How to use methods other than GET and POST in a web test 111
How to filter out certain dependent requests 111
How to handle ASP.NET Cookie-less Sessions 112
How to use Client-side certificates in web tests 112
How to remove the “If-Modified-Since” header from dependent requests 113
How to handle custom data binding in web tests 113
How to add a datasource value to a context parameter 113
How to test Web Services with Unit Tests 114
How to add random users to web tests 114
How to add think time to a Unit Test 114
How to add details of a validation rule to your web test 115
How to mask a 404 error on a dependent request 116
How to parameterize Web Service calls within Web Tests 117
How to pass Load Test Context Parameters to Unit Tests 117
How to create Global Variables in a Unit Test 117
How to use Unit Tests to Drive Load with Command Line Apps 118
How to add Console Output to the results store when running Unit tests under load 118
How to add parameters to Load Tests 119
How to Change the Standard Deviation for a NormalDistribution ThinkTime 119
How to programmatically access the number of users in Load Tests 120
How to create a webtest plugin that will only execute on a predefined interval 120
How to support Context Parameters in a plug-in property 121
How to stop a web test in the middle of execution 122
How To: Modify the ServicePointManager to force SSLv3 instead of TLS (Default) 122
How To: Stop a Test in the PreRequest event 123
How to make a validation rule force a redirection to a new page 123
How to add a Web Service reference in a test project 127
How to remotely count connections to a process 129
How to hook into LoadTest database upon completion of a load test 129
How to deploy DLLs with MSTEST.EXE 130
How to authenticate with proxy before the test iteration begins 131
How to enumerate WebTextContext and Unit TestContext objects 132
How to manually move the data cursor 132
How to programmatically create a declarative web test 133
How to modify the string body programmatically in a declarative web test 134
Gotcha: Check Your Validation Level in the Load Test Run Settings 134
Gotcha: Do not adjust goals too quickly in your code 134
Gotcha: Response body capture limit is set to 1.5 MB by default 134
Gotcha: Caching of dependent requests is disabled when playing back Web Tests 135
Best Practice: Blog on various considerations for web tests running under load 135
Best Practice: Coded web tests and web test plug-ins should not block threads 135
Best Practice: considerations when creating a dynamic goal based load test plugin: 136
Best Practice: Add an Analysis Comment 136
Best Practice – Using comments in declarative webtests 136
EXTENSIBILITY
New Inner-text and Select-tag rules published on Codeplex 138
ITEMS NOT SPECIFIC TO THE VSTS TESTING PLATFORM
Using the VSTS Application Profiler 146
VSTS 2008 Application Profiler New Features 146
Using System.NET Tracing to debug Network issues 146
Logparser tips and tricks 147
Logparser WEB Queries 147
LogParser Non-Web Queries 148
OLDER ARTICLES
Content-Length header not available in Web Request Object 149
SharePoint file upload test may post the file twice 149
Some Hidden Fields are not parameterized within AJAX calls 149
(FIX) Unit Test threading models and changing them 149
Bug in VSTS 2008 SP1 causes think time for redirected requests to be ignored in a load test 150
New Load Test Plugin Enhancements in VSTS 2008 SP1 150
Four New Methods added to the WebTestPlugin Class for 2008 SP1 150
INDEX
Ed.