Setting Up Machines and Collecting Diagnostic Information Using Test Settings
Test settings use diagnostic data adapters, which specify various types of data to collect when you run manual tests, automated tests, or both. Diagnostic data adapters can also specify how to affect the test machine. For example, a diagnostic data adapter might create an action recording, an action log, a video recording, or collect system information. Additionally, diagnostic data adapters can be used to simulate potential bottlenecks on the test machine or reduce the available system memory. For example, you can emulate a slow network to impose a bottleneck on the system.
Test settings define the following:
The type of tests you will run (manual or automated).
The set of roles that are required for your application under test.
The role to use to run your tests.
The diagnostic data adapters to use for each role.
Note
When you create test settings for your tests by using Microsoft Visual Studio 2010, you can run only automated tests. Therefore, you cannot select the type of tests to run. These test settings files that you create for Microsoft Visual Studio 2010 cannot be used with Microsoft Test Manager.
Running Tests by Using Microsoft Test Manager
If you run your tests using Microsoft Test Manager, you can configure a test plan to use a test setting for all its manual tests, and a test setting for all its automated tests if you need it. In addition, you can select an environment that includes the set of roles in your test settings for all the manual tests, and you must select an environment for all the automated tests in your plan. If necessary, you can override these test settings and environments when you run your tests.
Note
If you run automated tests by using Microsoft Test Manager, you must use an environment.
You configure a test plan by using the Properties page of the test plan in Microsoft Test Manager. The following illustration shows examples of test settings and environments for a test plan.
.png "Test Settings Concepts")
Note
Manual tests are always run on a local machine by using Test Runner. For more information, see Running Manual Tests Using Test Runner.
Running Tests by Using Microsoft Visual Studio
If you want to run your tests by using Microsoft Visual Studio 2010, you can configure the test settings to use when you run your tests. If you want to run your tests, collect data, or affect a test machine remotely, you must specify a test controller to use in your test settings. The test controller will have agents that can be used for each role in your test settings. For more information about test controllers and test agents, see Setting Up Test Machines to Run Tests or Collect Data.
Note
Environments are used only when you run your tests by using Microsoft Test Manager and not with Microsoft Visual Studio 2010.
Additional Information about Environments
You can create physical or virtual environments by using Microsoft Test Manager. A physical environment might use physical computers or virtual machines. A virtual environment uses only virtual machines. An environment consists of a set of roles. A role specifies the purpose of one or more computers in the environment. For example, a specific role could be "Web Server". You can run tests, collect data, or affect a test machine for each specific role in your environment.
You must specify a test controller for your environment. Each computer with a test agent registered to that controller is called a machine and is available to use for your environment. For more information, see Setting Up Test Machines to Run Tests or Collect Data.
The following illustration shows how you define the set of roles for your test settings. You can then select an environment that has machines that are assigned to each role to use when you run your tests. You can select any environment that includes at least the set of roles that are defined in your test settings. The environment may include other roles that are not specified in your test settings, as shown in the following illustration.
.png "Test Settings and Environments")
Test Settings Configuration Details for Diagnostic Data Adapters
The following table provides an overview of the various ways that the diagnostic data adapters can be configured for use with local or remote machines.
Diagnostic data adapter that is used in test setting |
Manual Tests on local machine |
Automated Tests |
Manual Tests: Collecting data by using a set of roles and an environment |
Notes |
|---|---|---|---|---|
Actions: You can create a test setting that collects a text description of each action that is performed during a test. When you configure this adapter, the selections are also used if you create an action recording when you run a manual test. The action logs and action recordings are saved together with the test results for the test. You can play back the action recording later to fast-forward through your test, or you can view the action log to see what actions were taken. |
Yes |
No |
Yes (See Notes) |
|
ASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy lets you collect information about the http calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters. |
Yes |
Yes |
Yes |
|
ASP.NET profiler: You can create a test setting that includes ASP.NET profiling, which collects performance data on ASP.NET Web applications. |
No |
Yes (See Notes) |
No |
|
Code coverage: You can create a test setting that includes code coverage information that is used to investigate how much of your code is covered by tests. |
No |
Yes (See Notes) |
No |
|
IntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This creates an IntelliTrace file that has an extension of .iTrace that contains this information. When a test fails, you can create a bug. The IntelliTrace file that is saved together with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file the local session can be simulated on another computer. This reduces the risk of a bug being non-reproducible. For more information, see Debugging with IntelliTrace. |
Yes |
Yes |
Yes |
|
Event log: You can configure a test setting to include event log collecting, which will be included in the test results. |
Yes |
Yes |
Yes |
|
Network emulation: You can specify that you want to place an artificial network load on your test by using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up. .gif "Note") Note
Network emulation cannot be used to increase the network connection speed.
|
Yes |
Yes |
Yes |
You can use the network emulation diagnostic data adapter for a client or server role. You do not have to use the adapter on both these roles that communicate with each other. If you include the network emulation diagnostic data adapter in the test settings and you intend to use it on your local machine, then you must also bind the network emulation driver to one of your machine’s network adapters. The network emulation driver is required for the network emulation diagnostic data adapter to function. The network emulation driver is installed and bound to your adapter in two ways:
Tip
You can also install the network emulation driver from the command line on your local machine without installing the Visual Studio test agent by using the following command: VSTestConfig NETWORKEMULATION /install
.gif "Caution note") Caution
The Network Emulation adapter is ignored by load tests. Instead, load tests use the settings that are specified in the network mix of the load test scenario. For more information, see Specifying Virtual Network Types in a Load Test Scenario and How to: Create a Test Setting for a Distributed Load Test.
|
System information: A test setting can be set up to include the system information about the machine on which the test is run. |
Yes |
Yes |
Yes |
|
Test impact: You can collect information about which methods of your applications code were used when a test case was run. This can be used together with changes to the application code that was made by developers to determine which tests were affected by those development changes. |
Yes |
Yes |
Yes |
|
Video Recorder: You can create a video recording of your desktop session when you run a test. The video can help other team members isolate application issues that are difficult to reproduce. |
Yes |
Yes (See Notes) |
Yes |
|
Note
You can also create custom diagnostic data adapters to fulfill a specific requirement that you have. For more information, see Creating a Diagnostic Data Adapter to Collect Custom Data or Affect a Test Machine.
Tip
Test Attachment Cleaner
The data that some of the diagnostic data adapters capture can use a lot of database space over time. The administrator of the database used for Visual Studio 2010 cannot control what data gets attached as part of test runs. For example, there are no policy settings that can limit the size of the data captured and there is not a retention policy to determine how long to hold this data before initiating a cleanup. You can use the test attachment cleaner tool to:
-
Determine how much database space each set of diagnostic data captures is using.
-
Reclaim the space for runs that are no longer relevant from a business perspective.
For more information and to download the test attachment cleaner tool, see Test Attachment Cleaner for Visual Studio Ultimate 2010 & Test Professional 2010.
Key Tasks
Use the following topics to help you create and configure test settings.
Tasks |
Associated Topics |
|---|---|
Create test settings to use with manual tests: You can create a test setting for a test plan by using the supported diagnostic data adapters for manual tests. |
|
Create test settings to use with automated tests: You can create a test setting for a test plan by using the supported diagnostic data adapters for automated tests. |
|
Update test settings: You can make corrections to existing test settings by editing them in the Lab Center or from the Properties activity for a test plan. |
|
Choose existing test settings for a test plan: Test settings are stored in Team Foundation Server and can be selected for use in multiple test plans. |
|
Collect your own data or affect a test machine by using custom diagnostic data adapters: You can create and use a custom diagnostic data adapter to perform tasks such as reduce available system memory, reduce available disk space, or allocate and hold other resources during a test. |
Related Tasks
Running Manual Tests Using Test Runner
You can create manual tests by using Microsoft Test Manager and run the tests by using Test Runner to record whether each step passes or fails. You can save the test outcome and any data that is collected when you run the test.
You can run tests directly from Microsoft Visual Studio 2010, from Team Foundation Build, or from the command line. You can use mstest.exe to run your automated tests from the command line, or you can use tcm.exe to import your test methods into test cases. You can then run the test cases for specific configurations from the command line or Microsoft Test Manager and save the results for the appropriate test plan.
See Also
Tasks
How to: Create a Test Setting for a Distributed Load Test