How to: Run Test Cases with Automation from the Command Line Using Tcm

  • Article

You can run test cases with associated automation from the command line using tcm.exe instead of using the user interface provided by Microsoft Test Manager. This enables you to start runs automatically from a batch file. For example, you can then decide to schedule that batch file to run every time that a build is completed.

To run these automated test cases so that the test results are part of your test plan, you must use a physical or virtual environment. When you start the run from the command line, use tcm.exe. You must specify both the test settings that contain the set of roles that you require to run your tests, and an environment that contains this set of roles. For more information about test settings and environments, see Setting Up Test Machines to Run Tests or Collect Data.

Note

If you want to run automated tests that interact with the desktop, you must set up your agent to run as a process instead of a service. For more information, see How to: Set Up Your Test Agent to Run Tests that Interact with the Desktop.

You can use a query to specify the test case and configuration pairing, which are named test points, or you can specify the test suites and configurations that you want to run from your test plan. You have to know the IDs for your test plan, your test suites, and configurations. You can use tcm.exe to list these for you if it is required.

Use the following procedures to gather the information that you have to have in order to run your automated tests from the command line and then run the tests:

  • List all the configurations in your team project

  • List all the test plans in your team project

  • List all the test environments for your team project

  • List all the test suites in your test plan

  • Create a test run using a test point query

  • Create a test run for a test suite and a specific configuration

  • View and update the test results

Common Parameters for the Procedures

Parameter

Description

/collection:CollectionURL

Specifies the URI of the team project collection. The format for the URI is as follows: http://ServerName:Port/VirtualDirectoryName/CollectionName

If no virtual directory is used, then the format for the URI is as follows:

http://ServerName:Port/CollectionName

If you do not know the correct path, contact your Team Foundation system administrator.

/teamproject:TeamProjectName

The team project that you want to use. This team project must be defined in the project collection specified by the /collection parameter.

/login:UserName,Password

Optional. Specifies the name and password of a user who is logged on to the application-tier server for Team Foundation Server and who has permissions to run the command.

You would use this option if your Windows credentials do not have the appropriate permissions, or you are using basic authentication, or you are not connected to a domain.

/? or help

Displays help about the command in the Command Prompt window.

Procedures

List All the Configurations in Your Team Project

To list all the configurations in your team project

  1. Open a Visual Studio command prompt.

    To do this, click Start, point to All Programs, point to Microsoft Visual Studio 2010, point to Visual Studio Tools, and then click Visual Studio Command Prompt (2010).

    The command prompt opens to the folder: <drive>:\Program Files\Microsoft Visual Studio 10.0\VC

  2. To list all the configurations in your team project, type the following at the command prompt:

    tcm configs /list /collection:<CollectionURL> /teamproject:<Team Project name>

    You will see results similar to these:

    Id Name

    ---------------- ------------------

    1 Vista and IE 7

    25 Vista and IE 8

    Note

    You can list plans for your team project too by using the following command: Tcm plans /list /collection:<CollectionURL> /teamproject:<Team Project name>

List All the Test Plans in Your Team Project

To list all the test plans in your team project

  1. Open a Visual Studio command prompt.

    To do this, click Start, point to All Programs, point to Microsoft Visual Studio 2010, point to Visual Studio Tools, and then click Visual Studio Command Prompt (2010).

    The command prompt opens to the folder: <drive>:\Program Files\Microsoft Visual Studio 10.0\VC

  2. To list all the plans in your team project, type the following at the command prompt:

    tcm plans /list /collection:<CollectionURL> /teamproject:<Team Project name>

    You will see results similar to these:

    Id Name

    ---------------- ------------------

    1 Beta 1 Test Plan

    22 RTM Test Plan

  3. (Optional) To find all the test suites in a test plan, type the following at the command prompt:

    tcm suites /list /planid:<your plan id> /collection:<CollectionURL> /teamproject:<Team Project name>

    You will see results similar to these:

    Id Name

    ---------------- ------------------

    1 Priority 1 Test Suite

    15 Shopping Cart Test Suite

List All the Test Environments for Your Team Project

To list all the test environments for your team project

  1. Open a Visual Studio command prompt.

    To do this, click Start, point to All Programs, point to Microsoft Visual Studio 2010, point to Visual Studio Tools, and then click Visual Studio Command Prompt (2010).

    The command prompt opens to the folder: <drive>:\Program Files\Microsoft Visual Studio 10.0\VC

  2. To list all the plans in your team project, type the following at the command prompt:

    tcm testenvironments /list /collection:<CollectionURL> /teamproject:<Team Project name>

    You will see results similar to these:

    Name Description

    ---------------- ------------------

    Staging Environment Use to test staging environment

    Testing Environment for Customer ABC

List All the Test Suites in Your Test Plan

To list all the test suites in your test plan

  1. Open a Visual Studio command prompt.

    To do this, click Start, point to All Programs, point to Microsoft Visual Studio 2010, point to Visual Studio Tools, and then click Visual Studio Command Prompt (2010).

    The command prompt opens to the folder: <drive>:\Program Files\Microsoft Visual Studio 10.0\VC

  2. To find all the test suites in your test plan, type the following at the command prompt:

    tcm suites /list /planid:<your plan id> /collection:<CollectionURL> /teamproject:<Team Project name>

    You will see results similar to these:

    Id Name

    ---------------- ------------------

    1 Priority 1 Test Suite

    15 Shopping Cart Test Suite

Create a Test Run Using a Test Point Query

To create a test run using a test point query

  1. Open a Visual Studio command prompt.

    To do this, click Start, point to All Programs, point to Microsoft Visual Studio 2010, point to Visual Studio Tools, and then click Visual Studio Command Prompt (2010).

    The command prompt opens to the folder: <drive>:\Program Files\Microsoft Visual Studio 10.0\VC

  2. To create a test run that uses a test point query for a specific test configuration, use the information in the following table.

    Id

    Value

    Planid

    1

    Configid

    13

    Type the following at the command prompt:

    tcm run /create /title:"Nightly Run" /planid:1 /querytext:"SELECT * FROM TestPoint WHERE ConfigurationId=13" /settingsname:"<Name of your automated test settings>" /testenvironment:"<Name of a matching environment>" /collection:<CollectionURL> /teamproject:<Team Project name>

    You will see results similar to these:

    Run created with ID: <xxx>.

    Note

    You must provide a test settings and an environment that matches the set of roles in your test settings for an automated test run using tcm.exe. For more information about test settings and environments, see Setting Up Test Machines to Run Tests or Collect Data. To create test settings, see Create Test Settings for Automated Tests as Part of a Test Plan.

  3. To view the progress of your test run, follow the steps in View and Update the Test Results.

Create a Test Run For a Test Suite and a Specific Configuration

To create a test run for a test suite and a specific configuration

  1. Open a Visual Studio command prompt.

    To do this, click Start, point to All Programs, point to Microsoft Visual Studio 2010, point to Visual Studio Tools, and then click Visual Studio Command Prompt (2010).

    The command prompt opens to the folder: <drive>:\Program Files\Microsoft Visual Studio 10.0\VC

  2. To create a test run for all the tests in a test suite for a specific configuration, use the information in the following table.

    Id

    Value

    Planid

    1

    Suiteid

    5

    Configid

    13

    Type the following at the command prompt:

    tcm run /create /title:"Nightly Run" /planid:1 /suiteid:5 /configid:13 /settingsname:"<Name of your automated test settings>" /testenvironment:"<Name of a matching environment>" /collection:<CollectionURL> /teamproject:<Team Project name>

    You will see results similar to these:

    Run created with ID: <xxx>.

    Note

    You must provide a test settings and an environment that matches the set of roles in your test settings for an automated test run using tcm.exe. For more information about test settings and environments, see Setting Up Test Machines to Run Tests or Collect Data. To create test settings, see Create Test Settings for Automated Tests as Part of a Test Plan.

  3. To view the progress of your test run, follow the steps in View and Update the Test Results.

View and Update the Test Results

To view and update the test results

  1. Open Microsoft Test Manager.

    Note

    To display the Microsoft Test Manager window, click Start, and then click All Programs. Point to Microsoft Visual Studio 2010 and then click Microsoft Test Manager 2010.

  2. To view the test results, click the down-arrow on the center group switcher and click Testing Center.

  3. On the center group menu bar, click Test and then click Analyze Test Runs.

    The Analyze Test Runs activity is displayed that shows any test runs for this test plan.

  4. Double-click the test run with the title that you gave it from the command line to open and view the details.

    The test run details are displayed.

  5. (Optional) To update the title of your test run, type the new name in Title.

  6. (Optional) If your test failed, you can update the reason for the failure. Click Resolution and select the reason for the failure from the list.

  7. (Optional) To add comments to the test result, click the Comments icon. Type your comments and then click Save comments.

  8. (Optional) To view the details of an individual test, double-click the test.

    The test result is displayed that shows the details from the test run and the test results history for that test. You can close this view to return to the test run.

    Note

    If you determine that there is a bug from your analysis, you can create a bug from this view.

  9. To save these changes for this test run, click Save in the toolbar.

See Also

Reference

tcm: Listing test plans, test suites, test configurations, and environments

tcm: Running Tests from a Test Plan Using the Command Line Utility

Concepts

Running Automated Tests from the Command Line

Running Automated Tests

Other Resources

Create Test Cases from an Assembly of Automated Tests