Create Test Settings for Automated Tests as Part of a Test Plan

Test settings use diagnostic data adapters, which specify various types of data to collect or how to affect the test machine when you run manual tests, automated tests, or both. For example, a diagnostic data adapter might create an action recording, an action log, a video recording, or collect system information. Additionally, diagnostic data adapters can be used to simulate potential bottlenecks on the test machine or reduce the available system memory. For example, you can emulate a slow network to impose a bottleneck on the system.

Test settings define the following:

• The type of tests that you will run (manual or automated)

• The set of roles that are required for your application under test

• The role to use to run your tests

• The diagnostic data adapters to use for each role

To run automated tests as part of a test plan, you associate your automated tests with a test case.

You can associate the automated test with your test case by using Visual Studio. You cannot make this association by using Microsoft Test Manager. You must first open the test case using Visual Studio as shown in the following illustration. For more information about how to run automated tests from a test plan, see How to: Run Automated Tests from a Test Plan Using Microsoft Test Manager.

Then you can associate the test method with your test case as shown in the following illustration.

You can also use a command line tool to create test cases from an assembly of automated tests. For more information, see How to: Create Test Cases from an Assembly of Automated Tests Using tcm.exe.

If you want to run automated tests as part of a test plan, you must select a set of roles for your test settings and use an environment that contains this set of roles in your test plan. You can add any roles that you require when you add an environment. For more information about roles and environments, see Setting Up Test Machines to Run Tests or Collect Data.

Use the following procedure to define test settings for automated tests that are part of your test plan and select a matching environment.

Creating test settings for automated tests as part of a test plan

To create test settings for automated tests as part of a test plan

1. Open Microsoft Test Manager.

   Note

   To display the Microsoft Test Manager window, click Start, and then click All Programs. Point to Microsoft Visual Studio 2010 and then click Microsoft Test Manager 2010.

2. Click the down-arrow on the center group switcher and then click Testing Center.

3. On the center group menu bar, click Plan and then click Properties.

   The properties for the selected test plan are displayed.

4. Click the drop-down arrow next to Test settings under Automated runs and then click New.

   The New Test Settings page is displayed with the General page selected.

   Note

   You can also create test settings in Lab Center by clicking Test Settings on the center group menu bar, and then clicking New.

5. Under Name, type the name for the test settings.

6. (Optional) Under Description, type a description for the test setting so other team members know what it is intended for.

7. Under What type of tests do you want to run, select Automated and then click Next.

   The New Test Settings page is displayed with the Roles page selected.

   Note

   If you are running automated tests, you cannot select the set of roles called Local to run locally because you must run the automated tests using an environment.

   For information about how to run a manual test, see Create Test Settings for Manual Tests.

   

8. Select a set of roles from the list that shows Sets of roles and Matching environments. Verify that there is at least one matching environment for your set of roles. If no matching environment exists, you might want to create an environment or modify an existing one. For more information, see Setting Up Test Machines to Run Tests or Collect Data.

   Warning

   You cannot add a role from the Test Settings Manager. If there are no roles that match what you have to have for your application, you must create an environment that includes those roles. When you create an environment, you can add roles.

9. From the drop-down list under Select the role to use to run your automated tests, select the role that you want to use to run your tests. Then click Next.

   The New Test Settings page is displayed with the Data and Diagnostics page selected.

   

10. To select the data and diagnostics that you want to collect for each role, select the role. For each role, select the diagnostic data adapters according to the needs of the tests in your test plan. To configure each diagnostic data adapter that you have selected for each role, click Configure.

    For detailed information about each diagnostic data adapter and how to configure it, you can view the associated topic in the following table.

    Note

    The table only shows the adapters that can be used with automated tests. For more information about diagnostic data adapters, see Setting Up Machines and Collecting Diagnostic Information Using Test Settings.

    Diagnostic Data Adapters for Automated Tests

    Diagnostic data adapter	Associated topic
    ASP.NET Client Proxy for IntelliTrace and Test Impact: This proxy allows you to collect information about the HTTP calls from a client to a Web server for the IntelliTrace and Test Impact diagnostic data adapters.	No configuration is required to collect this information.
    Event log: You can configure a test setting to include event log collecting, which will be included in the test results.	How to: Configure Event Log Collection Using Test Settings
    IntelliTrace: You can configure the diagnostic data adapter for IntelliTrace to collect specific diagnostic trace information to help isolate bugs that are difficult to reproduce. This creates an IntelliTrace file that has an extension of .iTrace that contains this information. When a test fails, you can create a bug. The IntelliTrace file that is saved with the test results is automatically linked to this bug. The data that is collected in the IntelliTrace file increases debugging productivity by reducing the time that is required to reproduce and diagnose an error in the code. From this IntelliTrace file the local session can be simulated on another computer, this reduces the possibility of a bug being non-reproducible. For more information, see Debugging with IntelliTrace.	How to: Collect IntelliTrace Data to Help Debug Difficult Issues
    Network emulation: You can specify that you want to put an artificial network load on your test using a test setting. Network emulation affects the communication to and from the machine by emulating a particular network connection speed, such as dial-up. Note   Network emulation cannot be used to increase the network connection speed.	How to: Configure Network Emulation Using Test Settings
    System information: A test setting can be set up to include the system information about the machine that the test is run on. The system information is specified in the test results by using a test setting.	No configuration is required to collect this information.
    Test impact: You can collect information about which methods of your applications code were used when a test case was running. This can be used together with changes to the application code made by developers to determine which tests were impacted by those development changes.	How to: Collect Data to Check Which Tests Should be Run After Code Changes
    Video Recorder: You can create a video recording of your desktop session when you run an automated test. This can be useful to view the user actions for a coded UI test. The video can help other team members isolate application issues that are difficult to reproduce.	How to: Record a Video of Your Desktop as You Run Tests Using Test Settings

11. Click Next.

    The New Test Settings page is displayed with the Advanced page selected. You can configure advanced settings for your automated tests if you have to have them.

12. To configure which directory is used to run your tests and to add files or directories to use to run your tests, click Deployment.

    1. To add a file to the directory that you are using to run your tests that you have to have for the tests, click Add file and then select the file that you want to add.

    2. To add a directory to the directory that you are using to run your tests that you have to have for the tests, click Add directory and then select the directory that you want to add.

    Note

    For more information about how to deploy files and directories for individual tests using properties and the DeploymentItem attribute, see How to: Configure Test Deployment.

13. To run scripts before and after your tests, click Scripts.

    1. Type the location of the script file in Setup script to run before the test run is started or click Browse to locate the setup script.

    2. Type the location of the script file in Cleanup script to run after the test run is completed or click Browse to locate the setup script.

14. To run your tests using a different host, click Hosts.

    1. To run your unit tests in the same process as an ASP.NET site, select ASP.NET in Host type. Then click Configure. For more information about how to configure the host, see Unit Tests for ASP.NET Web Services.

    2. Use the Run test in 32 bit or 64 bit process to select if you want your test to run as 32 bit or 64 bit process.

       Tip

       For maximum flexibility, you should compile your test projects with the Any CPU configuration. Then you can run on both 32 and 64 bit agents. There is no advantage to compiling test projects with the 64-bit configuration.

    3. Under For test that cannot be run on the specified host, select either Run in default host or Don't run.

15. (Optional) To limit the period of time for each test run and individual tests, click Timeouts.

    1. To abort a test run when a time limit is exceeded, select Abort a test run if the total time exceeds and then type a value for this limit.

    2. To fail a specific test if a time limit is exceeded, select Mark an individual test as failed if its execution time exceeds, and type a value for this limit.

16. (Optional) to apply unit test and Web performance test add-in options, click Add-ins.

17. (Optional) If you need to specify assembly locations that your unit tests need to load, click Configure associated with the Unit Test option.

    The Configure Add-in - Unit Test dialog box is displayed.

    1. For Root folder for the assemblies to be loaded, click Browse to locate the folder and populate the text box.

       The root folder that is specified can contain environment variables and represents the directory which will be used as the ApplicationBase of the AppDomain that the tests are run in. All the assemblies in this directory will be loadable by your unit tests. In a production environment, a good practice is to set this to the directory where your code under test assemblies are installed. In a development environment a good practice is to set this to the directory where your code under test assemblies are built to. This makes sure that any references that you have to the product binaries can be loaded and resolved during the discovery and execution of the tests without the need to copy the product binaries around with the tests.

       If no value is set here the ApplicationBase of the AppDomain that the tests are run in is set to the directory which contains the tests.

    2. Select or clear the check box for Use the Load Context for assemblies in the test directory.

       By default, most assemblies are loaded into the correct "Load Context". Usually, you should leave Use the Load Context for assemblies in the test directory checked. However, there are some conditions when you may want to turn this off. If there are a large number of assemblies in your test directory, and you have specified a location under Root folder for the assemblies to be loaded, and your tests are not dependent on being loaded in the Load Context, you could see a performance increase if you do not use the Load Context to load these test assemblies. If your tests depend on being loaded in a context other than the Load Context (not typical).

       For more information, see Best Practices for Assembly Loading.

    3. Under Folders to use when the tests are run, click Add folder.

       The Browse For Folder dialog box is displayed.

    4. Locate the folder to use and click OK.

       The Folders to use when the tests are run setting is the setting that you will probably use the most frequently. You can specify multiple paths to folders that assemblies should be resolved from during discovery and execution of the tests. Each of the paths that are specified in this section can contain environment variables. Together with each of the paths that are specified here, there are two options that are associated with it:

       First option   Select the Use Load Context check box to specify that the directory should use the load context when resolving assemblies from the directory (if the load context is not required for the tests to run correctly you may see a performance improvement by clearing this check box).

       Second option   Select the Include sub-folders check box to specify using any sub-folder to include when resolving assemblies from the directory.

    5. Under Additional folders to use when discovering tests, click Add Folder.

       The Browse For Folder dialog box is displayed.

    6. Locate the folder to use and click OK.

       This Additional folders to use when discovering tests is useful when you are either executing the tests remotely under Team Build or doing an automated run from Microsoft Test Manager. The paths provided here will be used for assembly resolution, but only during test discovery. These paths can contain environment variables. When tests are being scheduled to execute remotely from a build drop and not all the dependencies of the test assembly are in the same directory, these paths can be used to make sure that MSTest or the Test Controller can find enough of the dependent assemblies to discover the tests and schedule them to the remote machines for execution.

       For runs being scheduled from Microsoft Test Manager, there is an additional token "%BuildDrop%" that can be used to generically refer to the build drop location. This eliminates the need to create or update a Test Settings each time a new build is being tested. Unfortunately this token is not directly supported through Team Build (however if the build drop location is set in an environment variable named BuildDrop from the build definition it will have the same result).

       For more information, see Verifying Code by Using Unit Tests.

    7. Click Save.

    8. Click Close.

18. (Optional) To configure properties that control how Web performance tests are run in the test setting, click Configure associated with the Web Test option.

    The Configure Add-in - Web Test dialog box is displayed.

    1. Select either Fixed run count, or One run per data source row.

    2. Use the Browser type drop-down list to select the Web browser to use with your Web performance test. For example, Internet Explorer 8.0.

       For more information about Web performance tests, see Testing Application Performance and Stress.

       Note

       Web performance test requires Visual Studio 2010 Ultimate.

    3. Click Save.

    4. Click Close.

19. To show the summary for your test settings, click Next.

20. To save the test settings, click Save and Close.

21. A matching test environment is automatically selected in Test environment. If there are multiple test environments that match the set of roles in your test settings, you can select a different matching environment.

22. You can apply your changes for the test plan by clicking Save on the toolbar.

    Note

    If you have to change your test settings, click Open next to the Test settings drop-down arrow, or you can open the Lab Center, click Test Settings on the center group menu bar, and then click Open. For more information, see How to: Edit an Existing Test Setting for a Test Plan.

See Also

Tasks

Create Test Settings for Manual Tests

Create Test Settings to Run Automated Tests from Visual Studio

Concepts

Setting Up Machines and Collecting Diagnostic Information Using Test Settings

Setting Up Test Machines to Run Tests or Collect Data