How to run automated tests on different machines in parallel?
For this post, I am assuming that the tests are stored in test case management store of Tfs and you are queuing the test run from the tcm clients (Either MTM, TCM.exe or lab workflow). In case your tests are only in VS and you are running them via VS or MSTest, then you can control the distribution using the bucketing knobs (mainly bucket-size property, see appendix below for more details) available in VS.
To distribute the tests for a tcm run, you have following options.
Option 1 (Recommended one)
- Create multiple environments with each environment consisting of single machine (read single agent).
- Divide all your test cases onto different sets and queue one set on one environment and other set on other environment and so on.
- With this each set of test will run in parallel and on different machine.
Option 2
This option will work if you have less than 100 (default bucket size) tests in your test run.
- Create a single environment consisting of all the required machines and ensure that the machines where you want to run the tests, belongs to the same machine role. (In your test settings, you should specify this role as the execution role.)
- Change the MinimumTestsPerAgent property (see appendix below for more details) to 1 in the test controller configuration file (QTControllerConfig.exe.config). Make sure that you restart the test controller to bring this change in effect.
- Now queue the test run consisting of your <100 tests and you will notice that all the agents are busy in running the tests.
Option 3
This option will work if you have more than 100 tests in your test run.
- Create a single environment consisting of all the required machines and ensure that the machines where you want to run the tests, belongs to the same machine role. In your test settings, you should specify this role as the execution role.)
- Let us assume that you have 600 tests and have 3 machines to run them.
- Now in the test settings, change the bucketSize from default 100 to 200 (number of tests/number of machines i.e. 600/3). Please note that there is no gesture in MTM with which you can do this and will have to use this utilitywith following syntax.
UpdateTestSettings /collection:http:abc:8080/tfs/DefaultCollection /teamProject:myProject /settingsname:mysettings /bucketsize:200
- Now queue the test run and you will notice that all the agents are busy in running the tests.
Appendix: -
Bucket Size
Bucket size denotes the maximum size of each bucket that is assigned to an agent. Once the agents finishes the execution of a bucket of tests, it is given another one if more exist. The default value of the bucket size is 100. In VS it can be controlled by the Execution/Buckets/size attribute in the test settings as shown below.
<?xml version="1.0" encoding="UTF-8"?>
<TestSettings name="Local" id="d023da83-9635-4625-bcf9-fc8b0e92a9e1" xmlns=" https://microsoft.com/schemas/VisualStudio/TeamTest/2010" >
<Description>These are default test settings for a local test run.</Description>
<Deployment enabled="false" />
<Execution>
<Buckets size="100"/>
<TestTypeSpecific />
<AgentRule name="Execution Agents">
</AgentRule>
</Execution>
</TestSettings>
MinimumTestsPerAgent
This property is used for second level of bucketing and comes into picture when the number of tests that need to be distributed creates a partial bucket. For example if the number of tests are 20 (which is less than default bucket size 100) or 120 (in which you have 1 full bucket and other partial bucket with 20 tests), then for all partial buckets, this second level of bucketing is used.
Default value of this property is 10 and is controlled by “MinimumTestsPerAgent” app setting in controller’s configuration file (QTController.exe.config). This property is not present in the configuration file and you will have to add it.
<appSettings>
<add key="MinimumTestsPerAgent" value="1"/>
<add key="LogSizeLimitInMegs" value="20"/>
....
</appSettings>
Comments
Anonymous
August 12, 2010
Will this work for Coded UI Tests? For example, I have an environment with 10 machines. I have one Coded UI Test that I want to run on each machine in parallel.Anonymous
August 12, 2010
Yes, this will work for coded UI tests as well.So in your example, if you have 10 machines in an environment and 10 coded UI tests, then you can run 1 test on each machine in parallel.Anonymous
August 13, 2010
Aseemb,That's not my example. I have 1 (single) CodedUI test that I want to run on each machine in parallel. I don't have 10 seperate tests.I want to run the same single test on each machine in parallel.Anonymous
August 17, 2010
Brandon, Thanks for the clarification.This post only talks about distributing the tests across machines (like 5 tests to 2 machines) which can run in parallel and does not talk about running same test on n different machines.However for your scenario, you will have to either create 10 seperate test runs containing the single test each targetting different machine or in a single test run, add 10 different tfs test cases corresponding to single coded UI test method.Anonymous
November 02, 2011
The comment has been removedAnonymous
November 02, 2011
The comment has been removedAnonymous
November 03, 2011
The comment has been removedAnonymous
November 03, 2011
Suman,To achieve that, you will have to create environment with single agent and then queue32 separate runs. Run suite1 on environment1 (which contains agent1) and run suite2 on environment2 (which contains agent2). Queue suite3 on environment1 or 2 which will get picked up once that agent is done with its current suite.RegardsAseem BansalAnonymous
November 10, 2011
The comment has been removedAnonymous
February 13, 2012
Hi Aseem,Thanks for nice post.I did configuration as per Option 1. Now, when I try to run any test which is suppose to run in other machine, it is throwing exception:Unit Test Adapter threw exception:Could not load file or assembly 'eXimius.AutomatedTesting.WealthPlan, Version=1.0.0.0, Culture=neutral, PublicKeyToken=6676d32b6740bbf3' or one of its dependencies. The system cannot find the file specified..Other machine i have installed only Test Agent and register with controller. So, Is it require to do some more settings?Waiting for your reply...it is very urgent for meThanks,UmeshAnonymous
February 13, 2012
Umesh,Is the test working fine locally using mstest.exe?Also can you please check whether the mentioned assembly is deployed on the test agent machine or not?Typically the directory in which the tests are deployed on the agent machine is something like %LocalAppData of the user under which test agent is running% VSEQTQTAgent<Guid>Deployment.RegardsAseem BansalAnonymous
February 13, 2012
Aseem,Sorry! for late reply. Yes, Locally it is working fine. Actually, locally all dlls are registerd in GAC but in other machine we don't have VS2010 itself. only Test Agent.I verified that location which had suggested. Dlls were not deployed there so, manually I have added all required dlls to same location "C:Usersumesh.singhAppDataLocalVSEQTQTAgentf447527b-5b42-4bd8-aa6e-e351a380a04fTFBEMEV0268". Later I will add in Deployment setting of Lab center.I still not able to see it running inside other machine. Showing same error. Thanks,UmeshAnonymous
February 13, 2012
Thanks Aseem. After adding the deployment attribute in TestSetting of Lab Center, It started working fine.Anonymous
February 13, 2012
Hi Aseem,One strange issue I am facing. When I run one test(belong to other machine) from one MTM, It is executing 1st step of test case "opening application browser with URL". After this it is not doing anything. I guess, it might not recognizing UI elements. I am not sure.It is not logging any exception. So, unable to recognize the issue.Thanks,UmeshAnonymous
February 13, 2012
Good to know that the test worked fine for you.Regarding your new question, Can you please share whether you are running a "manual test" or something else?Also it will be great if you can post your question on the test manager form as someone more knowledgeable can reply to your question much sooner. Here is the link to the forum.social.msdn.microsoft.com/.../threadsRegardsAseem bansalAnonymous
February 13, 2012
I running automation test. I have one TestProject which are having so many CodedUI test cases. These test cases associated with Test Plans in MTM.I am going to place same question to test manager forum.Thanks for quick response.UmeshAnonymous
May 20, 2012
Hi Aseem,have you ever encountered the problem to get no code coverage results once the tests are executed in parallel?regards,mariaAnonymous
May 21, 2012
Hi Maria,As mentioned in the below msdn link, code coverage is not supported in remote runs as well as MTM based test runs. This means that without parallelism also, it must not be working for you.msdn.microsoft.com/.../dd504821.aspx >You can only use code coverage when you run an automated test, and only from the computer that is running the test. Remote collection is not supported. You cannot use code coverage when you run tests from a test plan using Microsoft Test Manager.RegardsAseem BansalAnonymous
July 19, 2012
Great info.. Thanks for the piece of code as well..The process seems to be very simple but only after executing it we will come to know about the difficulties..thanks a lot for the info related to testing on machines.. If people interested to know more details about various testing process on machines kindly visit http://www.datasysttest.com/Anonymous
October 17, 2012
The comment has been removedAnonymous
October 17, 2012
The comment has been removedAnonymous
April 25, 2013
Is possible to run same test on all test agents . Because we like to run the same test in different OS ( XP ,Win 7 and Win 8) for compatibility check of code ..is it possible .Anonymous
April 30, 2013
Jpohy, You should queue separate test runs, one for each OS.Anonymous
May 01, 2013
The comment has been removedAnonymous
May 04, 2013
Kal, Does it work now?Anonymous
August 05, 2013
Hi Aseem,I have the same requirement as what Brandon posted in the 1st post."I have 1 (single) CodedUI test that I want to run on each machine in parallel. I don't have 10 seperate tests.I want to run the same single test on each machine in parallel."Any idea how could that be achieved in parallel without having to create 10 test cases targeting a single test method? I have like 100+ test methods that would like to be re-run in 4 platforms and creating test cases solely for that purpose doesn't seems feasible. :(Anonymous
August 11, 2013
Boon,You can create n test runs containing the single Coded UI test each targetting different machine. Right?RegardsAseem BansalAnonymous
August 13, 2013
The comment has been removedAnonymous
August 13, 2013
Hi Kanchan,In which file have you added this bucket size?RegardsAseem BnasalAnonymous
August 13, 2013
The comment has been removedAnonymous
October 28, 2013
Can we run a same test case on different VMs simultaneously.Anonymous
October 31, 2013
Sandeep, If you have to run the same test case simultaneously on different VMs, then you will have to different test runs, each targeting a VM.RegardsAseem BansalAnonymous
January 24, 2014
The comment has been removedAnonymous
February 24, 2014
The comment has been removedAnonymous
February 24, 2014
With the option '1' Aseem mentioned, you still need to trigger of 'n' number of MTM test runs, or 'n' number of builds(of different definition) each using a different environment and running different tests. There is no way to use more than one environment in a single BDT or MTM test run.Having said that, you can still accomplish parallelism by using option '3', as you can create a single environment and prepare multiple machines where tests can run (you anyway have to do this for multiple environments as well).Anonymous
February 24, 2014
Thanks Diganta for a quick replyLet me explain my test scenarios in bit more detail.We have 500 cases approximately to be executed.These cases are grouped into different folders under the Test Suite.Among these cases are the Install & Configuration cases.The remaining cases are dependant on the above set of cases.So if we need to distribute the entire suite in 2 to 3 machines,the Install & Config cases will have to be run on all the three machines.I dont have a clue on how to achieve this part.Once this is done, I can then set the bucket size to 100 or 200 & run the cases simultaneously on the 3 setups.Anonymous
February 24, 2014
This is a blocking issue for us.Any help would be highly appreciable.Anonymous
September 09, 2014
The comment has been removedAnonymous
September 09, 2014
Prashant,You should do the following: -Import the test case to your test plan if they are not imported already.Select the above tests in the testing center (MTM -> Test) and do a run with options specifying the environment. You can find more details about this here.msdn.microsoft.com/.../dd286580(v=vs.110).aspxRegardsAseem BansalAnonymous
September 09, 2014
One of my colleague (Ravi) gave me a very good link about the flow.msdn.microsoft.com/.../dd380741.aspxPlease try this out and do let me know in case you have any further questions.RegardsAseem BansalAnonymous
September 14, 2014
Thanks a ton for quick response,Aseem, I have setv environment and already we have manual test cases in out test plan, can you suggest me how to import automation scripts in test plan.again, wuich response will be highly appreciated :)Anonymous
September 16, 2014
The msdn link which I have mentioned above contains the steps to import the automated tests in to test plan as well. Copy pasting the steps from the above link.RegardsAseem Bansal1.Open a Developer Command Prompt, and change directory to the output director of your Visual Studio solution.cd MySolutionMyProjectbinDebug2.To import all the test methods from the solution:tcm testcase /collection: CollectionUrl /teamproject:MyProject /import /storage:MyAssembly.dll /category:"MyIntegrationTestCategory"The category parameter is optional but recommended. You only want to create test cases from integration or system tests, which you can mark by using the [TestCategory ("category")] attribute.3.In the Test hub in Team Web Access or in Microsoft Test Manager, use Add Existing to add the test cases to a test suite.Anonymous
September 17, 2014
Is it necessary to configure build ID under build heading for automation run.?Anonymous
September 17, 2014
In addition to above comments, Automation run is throwing error "The build directory of the test run either does not exist or access permission is required." is it necessary to enter build as in the settings build is not required.Anonymous
September 18, 2014
The comment has been removedAnonymous
September 21, 2014
Thanks Aseem,I have done basic configuration and setup agent controller setup..I am getting error "cannot create automated test run. A build associated with the active test plan is required to submit an automated test run".Can you please suggest what is this error about|?would appreciate quick response as the issue is delaying my work,.Anonymous
September 21, 2014
Thanks for sharing this info, however we have multiple build ID and build is not locally placed. also I tried selecting specific build in filter for build dropdown there is no Build available under available build dropdown to assign this build to a plan...Is there a configuration issue that no build is available (as more likely it looks) or any configuration I am missing.What I did---> selected build id in test plan settings.--> Saved the configuration.--> Trying assigning the selected build to same plan, getting no build available in dropdown..Please suggest, may be this can be a weird/silly issue for you :) but it has now become a showstopper for us :) ..please suggest.Anonymous
September 21, 2014
Is it necessary to select build ID etc , I need to know as we are using/invoking URL(updated) in our scripts... Please let me know if we can skip this step and how?Anonymous
September 21, 2014
Prashant,You need to have a build which contains tests. Another user had faced similar issue, can you please check the instructions mentioned here works for you or not?social.msdn.microsoft.com/.../running-automated-tests-in-microsoft-test-managerRegardsAseem BansalAnonymous
September 26, 2014
The comment has been removedAnonymous
September 26, 2014
The comment has been removedAnonymous
September 30, 2014
The comment has been removedAnonymous
October 13, 2014
Hi Aseem,We are facing some issue while executing VSTS scripts in parallel.While running all scripts in automation manual run got started and autoamtion did succed in that case if we are running 1 automation script at a time from MTM then it is running.How can we runn all scripts using automation in one gowithout starting manual run.Please suggest.Anonymous
October 15, 2014
Jatin/Prashant, Are the tests running on your machine using mstest.exe? It looks like one of required assembly is missing in your build.Pankaj, Do you want to convert your manual test cases to coded UI tests?Jatin/Prashant/Pankaj, Can you please post your question on the forum for faster replies by the community?social.msdn.microsoft.com/.../homeRegardsAseem BansalAnonymous
October 28, 2014
It worked Aseem, Thanks a lot for your support....What all benefits do you think it provides as compare to executing on single machine.Anonymous
October 29, 2014
Prashant, The underlying engine is same in both the cases, so if it is possible for you to use your single machine you can continue doing that. But there are cases when you cannot use your machine because of resource constraints or time taken, then you would want framework to use the lab machine(s) and distribute the tests across machines so that they can complete as fast as possible.Anonymous
November 01, 2014
Hey Aseem. Is it possible to assign multiple builds for different environment.e.g. I have created 2 environment in MTM. One is Dev and other is Test environment. Now, i want the different build for Dev and different build for test. Is So that when i run the tests, i can select the build.Anonymous
November 04, 2014
Yes, You can do that. There is an action called "Run with options" in MTM with which you can specify the build # for the new test run.msdn.microsoft.com/.../dd553270(v=vs.110).aspxRegardsAseem BansalAnonymous
November 12, 2014
Hi Aseem, thanks for the clarification. I have created two environments. But the problem I am facing is that one environment works and if i run other environment, then "Waiting for Controller" is displayed under Analyze Test runs link. Only one environment is working. If i want to run both the environments then is there any settings in config file. Appreciate your inputs.Anonymous
December 07, 2014
I have 120 tests with the BucketSize = 100 and MinimumTestsPerAgent = 1 and out of the 120 tests the last 20 are data driven with each of them running say 30 iterations each. Effectively if I count each iteration as a test, I have 20 * 30 = 600 + 100 = 700 tests. How would the distribution work in this scenario?Anonymous
December 09, 2014
Found the answer myself. The bucket size has no regard to number of iterations a test can have (when data driven). It treats a test with one iteration the same a test with 100 iterations.I suppose Microsoft could do more here..1) Provide easy accessibility to settings like MinimumTestsPerAgent and BucketSize2) Implement more settings that will enable user to configure the test runs to make the most efficient use of the resources available.Anonymous
December 10, 2014
Thanks Syed. These are good suggestions. We have added these asks to our backlog for consideration.Anonymous
January 28, 2015
The comment has been removedAnonymous
January 28, 2015
HI SyedWe unfortunately do not have fine grained control on how to restrict/avoid distribution of tests within a run.Anonymous
January 28, 2015
Hi Syed,Have you explored ordered tests? Ordered tests will ensure that tests run in the specified order.Anonymous
May 05, 2015
Thank you so much for the information! Finally figured it out, thanx to you!Anonymous
May 10, 2015
Some tests run twice(cannot tell if on different machines or on the same machine). How come?Anonymous
August 11, 2015
Hi I have 30 Test scripts and 6 Machines. So lets assumes it will run 5 on each machine. I have written code to send result email in my Test Cleanup. Now every agent is sending emails so I am getting 5 emails. But i want one single email. Could you please tell me how can i implement it?Anonymous
August 18, 2015
Hi, I have two test scripts associated to 2 test cases in MTM. Also i have configured lab manager with one environment having 2 machines . I want to run both the scripts simultaneously on two machines . can you please let me know what other setting do i need to do so that i can run 1 TS(test Script) on 1 m/c and other TS on other m/c simultaneously ?Anonymous
February 28, 2016
The comment has been removedAnonymous
February 28, 2016
Hi Prachi Have you tried to invoke your send email from the cleanup script, which you can specify in your testsettings?