Test controller/agent usage recommendations

  • Article

Over the years we have had many people ask us on what is the best way to configure and use the test controller and agent for remote test execution. Here is a summary of the best practices that we advocate.

Setup and Settings

If you are on the latest VS2012U1+ or VS2013 you can ignore step#1&2

  1. Ensure that Visual Studio 2010 SP1 is installed on all machines

    Dev10 SP1 - https://www.microsoft.com/en-us/download/details.aspx?id=23691

  2. Next install the latest patch available

    https://support.microsoft.com/kb/2801364

  3. Open "<VSINSTALLDIR>\Common7\IDE\QTControllerService.exe.config" and add the following entry under

    <add key="ControllerJobSpooling" value="false"/>

    • Restart Controller

  4. Agent setup

    • Install the latest patch available
    • Open "<VSINSTALLDIR>\Common7\IDE\QTAgentService.exe.config" and add the following

    <add key="RestartTestExecutionProcessForEachRun" value="true"/>

    • Restart Agent

  5. Client Setup

    • Install the latest patch available
    • Open "<VSINSTALLDIR>\common7\ide\mstest.exe.config and add the following key

    <add key="DeleteTestDeploymentFiles" value="yes" />

  6. In your test settings

    • Keep bucket size as 1000. This will result in a test run going to smaller set of agents.

    <Execution>

        <Buckets size="1000"/>

  7. Set test run timeouts to ensure no runs go beyond acceptable limits

    <Execution>

    <Timeouts runTimeout="7200000" />

    7200000 = 2hrs specified in milliseconds

  8. Run your tests in parallel if your agents are on a multiproc system

    <Execution parallelTestCount=0>

    https://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx

Other Recommendations

  1. Test Controller keeps all results for a run in memory. Therefore it is necessary that test runs be split into logical groups. We recommend you split tests into assemblies so that each test dll has <=5000 tests and/or run the TC in 64bit mode
    1. testcontrollerconfig.exe configure <… configuration parameters… > /platform64
  2. When any component of the Test Rig (any of the agents or controller) faces an issues (network disconnect, deployment failures, test crashes), the current test run is aborted. The probability of the test run getting aborted due to failure in agents reduces when we send one test run to a smaller set of agents. Keep a run limited to minimum number of test assemblies
  3. To scale out you should be using multiple such rigs and breaking up your test runs amongst these rigs as required.

Comments

  • Anonymous
    December 03, 2013
    I can't find QTAgentService.exe.config. My <VSINSTALLDIR>Common7IDE has QTAgent, QTAgent_35, QTAgent_40, QTAgent32, QTAgent32_35, QTAgent32_40, QTDCAgent and QTDCAgent32.It might be important to not that the build agents aren't on the same machine as the build controller
  • Anonymous
    December 04, 2013
    Oh Test Controller/Agent and Build Controller/Agent isn't the same thing, right? That's my problem I think...