Executing Unit Tests in parallel on a multi-CPU/core machine

  • 项目

In Visual Studio 2010, we introduced the ability to run tests in parallel.  Many machines today have multiple CPU’s or a CPU with multiple cores, so it makes sense that we may want our tests to run in such a way to use all the cores we have available on our machine.  This will effectively increase the number of tests running at the same time, which will reduce the time to run all the tests.

Decisions

We looked at all the different test types and functionality that we have, and made the following decisions:

  1. The only test type that we would support for multi-CPU/core execution is the unit test.  The decision was based on various things such as: Is the test type doing UI testing? Is the test already doing some form of parallel execution? Is the test type hosted in a different host adapter (we may not have control over how the tests are executed)?  Are tests part of another test which may ensure a certain order of execution?  Is the test a unit test extension? (we may not have control over how the test is executed)? The answer to all these questions led us to support only unit tests.
  2. Data Adapters cannot be enabled.  Since data collection is based on test events, having data collection while multiple tests are execution quickly could slow things down.  Also, the data itself may not correctly be isolated to the test specified, but would include data or information from other tests that are executing.
  3. This only occurs for local execution through Visual Studio or MSTest.  Since remote execution allows you to already divide up tests across multiple agents, we have decided not to enable this scenario for remote execution.  Also, this currently is only available in Visual Studio or MSTest.

Requirements

So, you ask, get on with telling me what I need to do!

Okay, but first we must remind you, and it also kind of goes without saying, but I will say it anyway; you need a machine with multiple cores.  Virtual machines can be used but you will need to make sure that the machine has multiple CPU’s.  An easy way to find out is to view the task manager.

image

My machine is single CPU, but has multiple cores.

Also, your tests must be thread-safe.  Only you can ensure that they are.  Failure to do so can result in incorrect results, deadlocks, and a lot of head-aches.  Take some time to review the thread-safe link and review your code for thread-safety.

How to: Enable parallel test execution

Okay, so now, how do we do it?  I will put all the steps from above into this list

  1. Ensure you have a multi-core/CPU machine (see above requirement).
  2. Ensure you are running only unit tests (see above decisions)
  3. Ensure your tests are thread-safe (see above requirement)
  4. Ensure you do not have any data adapters on (see above decisions)
  5. Ensure you are running locally (see above decisions)
  6. Modify your test settings file.
    Right-click the test setting file and select “Open With”
    image

    1. Open as Xml
      image

    2. Set the parallelTestCount attribute on the Execution element
      image 

      Options are:
      attribute not specified = 1 CPU/Core used (default)
      0 = Auto configure: We will use as many tests as we can based on your CPU and core count
      n = The number n of tests to run in parallel to use (if you do not want to use all of your CPU/cores)

    3. Save your settings… and you are done.

Result

I have two cores, and so here is my resulting run:

image

As you can see, we have two tests running at the same time.  If we were running this in serial, we would at least take 10 seconds.  With parallel tests we got this down to 6 seconds

image

Of course there are many other factors that affect this number.  There is the cost of starting and tearing down the run, so you will see a lesser effect if you have a few tests.  It also depends on the number of CPU/cores you have and of course how fast your tests execute.

Good luck and hope this helps speed things up for you.

Bruce Taimana
Program Manager
Visual Studio Team Test

Comments

  • Anonymous
    June 14, 2010
    Excellent feature.  With 6+ cores being cheaply available now, this will provide massive performance gains.  We use this feature with our ecommerce project that has thousands of tests, and our CI build time has been cut by a factor of 5.  Of course, the caveat is that your tests follow more strict TDD guidelines, but generally speaking, unit tests should be independent.  There are some issues if you doing any threading magic in the code being tested (as Bruce alludes to when he says your tests must be thread safe), but other than that it is awesome to be able to take advantage of multi-core/CPU setups this way.

  • Anonymous
    September 30, 2010
    Great feature for teams that have independant unit tests like us. The best point is to let the runtime determine the number of cores on the running machine. It is going to help us speed up the validation process during a deployment.

  • Anonymous
    October 19, 2010
    The comment has been removed

  • Anonymous
    October 28, 2010
    This is neat. Is it possible to parallelize a Data-Driven unit test in Visual Studio 2010?

  • Anonymous
    November 15, 2010
    The comment has been removed

  • Anonymous
    January 21, 2011
    The comment has been removed

  • Anonymous
    April 14, 2011
    The part that was missing was here: Step 6: Additionally (whether a bug or by design) the parallel test option is only read when the project is loaded. Therefore if you change the setting you'll have to close the project and reopen it before it'll take effect. Thanks

  • Anonymous
    October 09, 2012
    What about Visual Studio 2012? I cannot find these options.

  • Anonymous
    December 19, 2012
    You find them in the .testrunconfig file ...

  • Anonymous
    April 24, 2013
    Doesn't work with new VS2012 .runsettings? How do we run unit tests in parallel on VS2012?

  • Anonymous
    August 20, 2013
    Hello, I am trying to run data driven unit tests in parallel against Test controllerTest Agents. The tests are running fine. But as part of the tests we are logging to the Console.WriteLine. Some of the tests are throwing exceptions at the Console.WriteLine statements as below. My test code looks like: I have 5 tests similar to below one. [TestMethod] [DataSource("Microsoft.VisualStudio.TestTools.DataSource.XML", "|DataDirectory|\TestData.xml", "Test1", DataAccessMethod.Sequential)] public void MyTest1() { Console.WriteLine((System.Reflection.MethodBase.GetCurrentMethod().ToString())); string val1 = testContext.DataRow["val1"].ToString(); string val2 = testContext.DataRow["val2"].ToString(); Console.WriteLine("val1: " + val1); Console.WriteLine("val2: " + val2); System.Threading.Thread.Sleep(3000); } Error Message: Initialization method ParallelTestDataDriven.UnitTest1.TestInit threw exception. System.ObjectDisposedException: System.ObjectDisposedException: Cannot write to a closed TextWriter.. Stack Trace: System.IO.__Error.WriterClosed() System.IO.StringWriter.Write(Char[] buffer, Int32 index, Int32 count) Microsoft.VisualStudio.TestTools.TestTypes.Unit.ThreadSafeStringWriter.Write(Char[] buffer, Int32 index, Int32 count) System.IO.TextWriter.WriteLine(String value) System.IO.TextWriter.SyncTextWriter.WriteLine(String value) ParallelTestDataDriven.UnitTest1.TestInit() in I tried to work around using TraceSource class. With this only logging is happening for the first row(in a multi row test). There is no logging from the 2nd data row onwards. I tried log4Net. With  this: the logging is getting jumbled. For eg: log for data row 2 is shown in data row 3. I am blocked on this. Any help is appreciated.

  • Anonymous
    September 06, 2013
    Hello Can we run tests in parallel based on test categories? Eg: [TestCategory("1")] [TestMethod] public void Test1() {} [TestCategory("1")] [TestMethod] public void Test2() {} [TestCategory("3")] [TestMethod] public void Test3() {} I want to run all the tests which are having test category as "1" in parallel. Can you please let me know if there is a way.

  • Anonymous
    October 30, 2013
    Why was this feature removed in versions newer than 2010?

  • Anonymous
    July 15, 2015
    The comment has been removed

  • Anonymous
    January 23, 2016
    Good to know the capabilities. Thanks for sharing team!

  • Anonymous
    February 14, 2016
    Cool feature, looking forward to seeing more progress!