Use Your Build System to Work with Tests

You can define a build process that runs tests and analyzes the impact of code changes on your tests. For example, you can define a build process to serve as your team’s regularly scheduled build verification test (BVT) run.

In this topic

• Prepare to Run Tests in Your Build Process

• Work with Tests in a Build Based on DefaultTemplate.xaml

  • Required Permissions

  • Run Automated Tests

  • Temporarily Disable Tests

  • Enable Test Impact Analysis

• Work with Tests in a Custom Build Process

• For More Information

Prepare to Run Tests in Your Build Process

Prepare Your Tests: Make sure that the tests that you will run from your build process do not require user interaction. In general, tests that display a UI are not appropriate for your build system to run. If you must use your build system to run a test that displays a UI, you must run the build definition on a machine that is running interactively. For more information, see Configure a Build Machine.

Prepare Your Build Machine: Some kinds of tests can be run only by a build agent on a build machine that is specially configured. Before you try to use your build process to run tests, make sure they can be run on the build machine that you plan to use. For more information, see Use Your Build Agent to Run Tests.

Work with Tests in a Build Based on DefaultTemplate.xaml

You can design your build to run one or more sets of tests and to analyze the impact of code changes on your tests.

Required Permissions

To perform these procedures, your Edit Build Definition permission must be set to Allow. For more information, see Team Foundation Server Permissions.

Run Automated Tests

You can design your build to run one or more sets of tests. For each set, you can specify the following settings:

• which tests are run

• which settings are used to run the test

• whether the build should fail if a test fails

To configure your build to run automated tests

1. On the Process tab of your build definition, expand the Basic node.

2. Select the Automated Tests box, and then click the ellipsis button (...) in this box.

   The Automated Tests dialog box appears.

3. Perform one of the following steps:

   • To add a set of tests, click Add.

   • To modify a set of tests, click it, and then click Edit.

   The Add/Edit Test dialog box appears.

4. Click the method that you want use to run the tests:

   • Test assembly file specification (recommended)

     Leave the default value (**\*test*.dll) if you want the build agent to search recursively for any .dll files that match *test*.dll in the binaries subdirectory of the build agent's working directory. As an alternative, modify the file specification to meet your needs.

     (Optional) Specify a Test Settings File to configure how the tests are run. For more information, see Create Test Settings to Run Automated Tests from Visual Studio.

   • Test metadata file (.vsmdi)

     If you click this option, click Browse to locate and specify the test metadata file that you want to use. You can then either leave the Run all tests in this VSMDI file check box selected, or you can clear it and then select one or more tests in the Test lists to run list.

     For more information, see Defining Test Lists to Group Your Tests.

5. Click the Criteria/Arguments tab.

6. (Optional) Filter the tests to run by category.

   For more information, see Specify Test Categories later in this topic.

7. (Optional) Filter the tests to run by priority.

   Tip

   If you assign priorities to your tests, you can use this parameter to better balance thorough testing with faster builds.

   Set Minimum Test Priority to either a positive integer that is equal to or lower than Maximum Test Priority or to -1 to indicate no minimum value.

   Set Maximum Test Priority to either a positive integer that is equal to or higher than Minimum Test Priority or to -1 to indicate no maximum value.

8. If you want the build to fail if any of the tests in this set fail, select the Fail Build on Test Failure check box. If you leave this check box cleared and any test fails, the completed build will be classified as Partially Succeeded.

9. Click OK.

Specify Test Categories

You can use test categories to filter the set of tests that are run.

Categorize Your Tests

You can assign categories to your tests by using Test Professional. For example, you could create a test category called CI and then specify that category in your continuous integration builds. You could create another category for your build verification tests called bvt and then specify that category in scheduled builds, such as your nightly build.

For more information, see Defining Test Categories to Group Your Tests.

Specify Test Categories for a Set of Tests

You can specify test categories in the following ways:

• When you define the set of tests, as explained earlier in this topic.

• By modifying a set of tests. (You modify a set of tests by expanding the Basic node, expanding the Automated Tests node, expanding the set that you want to modify, and then clicking in the Category Filter box.)

You can specify test categories in one of the following forms:

• Specify a single test category to include or exclude. For example, you could have a test category that is called bvt. You would set this parameter to bvt to run only tests in this category or !bvt to run all tests except those tests in this category.

• Specify multiple test categories by using the & ("and" operator) and the ! ("not" operator). For example, you can specify quick&gui&!deep to run only those tests in both the quick and gui categories but not in the deep category.

• Specify multiple test categories by using the | ("or" operator) and the ! ("not" operator). For example, you can specify quick|gui|!deep to run tests in the quick category, tests in the gui category, and any tests that are not in the deep category.

Define Multiple Sets of Tests

You can define as many sets of tests as you need to meet the requirements of your team's build and test process. For example, you might need to define multiple sets of tests to run in a single build in the following scenarios:

• You have two sets of tests:

  • A set of top-priority core tests that must pass. You define a set of tests that includes a Minimum Test Priority and Maximum Test Priority of 1. You select the Fail build on test failure check box.

  • A set of less important tests that you want to run but that are not required to pass for the build to be usable. You define a set of tests that include a Minimum Test Priority of 2 and a Maximum Test Priority of 3. You leave the Fail build on test failure check box cleared.

• You want to run the same set of tests with different test settings.

• You want the main set of assemblies that you build to be subject to code coverage. However, you have another set of assemblies from an external source that do not require code coverage. To enable this kind of process, you could use two sets of tests that are configured to use two groups of test settings files.

Temporarily Disable Tests

If you must temporarily disable tests without deleting the test sets that contain them, expand the Advanced node, and set Disable Tests to True. Set the value back to False when you want to enable tests again.

Enable Test Impact Analysis

Your testers and developers may need to know how code changes that are encompassed in a completed build have affected your tests. When you enable test impact analysis in a build, the system analyzes and then reports on how code changes affected your tests in the build report of the completed build.

To enable test impact analysis

1. Configure test impact analysis in a test settings file.

   For more information, see How to: Collect Data to Check Which Tests Should be Run After Code Changes.

2. Create a set of tests that is configured to use the test settings file.

   For more information, see Run Automated Tests earlier in this topic.

3. Expand the Advanced node, and make sure that Analyze Test Impact is set to True and that Disable Tests is set to False.

Work with Tests in a Custom Build Process

You can run tests and analyze the impact of code changes on your tests from a custom build process. For more information, see Run Tests (MSTest Activity).

For More Information

Define a Build Using the Default Template provides more information about how to create a build definition that is based on DefaultTemplate.xaml.

How to: Configure and Run Scheduled Tests After Building Your Application provides more information about how to set up and run build verification tests (BVTs).

Build and Deploy Continuously provides more information about how to run tests as part of a continuous-integration strategy.

Guidance for Build, Deploy and Test Workflows describes the recommended approaches for your build, deploy, and test workflow.

See Also

Concepts

Define Your Build Process

Change History