Differences between MSTest.exe and VS IDE

  • [アーティクル]

MSTest is a good unit test solution , tightly integrated with Visual Studio IDE .Following is a typical flow for test automation development and running the tests

  1. Tests are developed and tested completely in VS IDE .
  2. MSTest.exe is used for running the tests in lab .

MSTest.exe is the command line test execution tool . Details about various options that it provides are here .

Mostly , MSTest.exe and VS IDE are similar . Following are some subtle differences

  1. If a project has other test projects referenced , VS IDE would run tests from all the referenced projects. Whereas MSTest.exe would only execute the tests that are directly in the dll provided in command line for /testcontainer parameter.
  2. VS IDE finds out all the necessary dlls required , it walks the depency graph and does this for us. MSTest.exe doesn't do this . This might result in a failure during automated runs,especially when the tests are run in remote machine using a Test Controller + Test Agent setup . Following are some solutions to this issue
    1. Make sure "Copy Local" option is set for all custom dlls that are referenced.
    2. Use Deployment Item option in TestSettings file to copy dlls that will not be available in the machine where tests will be run. Pass this test setting file as a parameter to MSTest.exe command line.
  3. VS IDE works with relative paths .MSTest.exe won't in most cases , because design time relative path and run time relative paths would most likely be different. This issue shows up mostly for load tests.Load tests are xml files , Unit tests that are part of loadtests could be specified using relative path or using absolute path. While developing the load test with VS IDE , it adds the tests using relative path. This may not work while running the load test from command line using MSTest.exe . Following are some possible solutions for this issue
    1. Provide a placeholder location during build time (like an environment variable) , and replace it during automation run time. For example the location could be "%BuildRoot%\TestDll.dll" at design time and changed to "c:\CurrentBuild\TestDll.dll" at run time.