Configuring Playback in Visual Studio 2010

In a previous post, Abhishek had talked about the Record and Playback Engine in VSTT 2010.  The Coded UI Test feature in Visual Studio 2010 also uses this engine to playback UI actions.  In this post, i will describe the various settings for Playback and how you can modify them to improve the resilience of your Coded UI Tests. Playback is configured by modifying the fields in Playback.PlaybackSettings class.

1. Continue on Error

When you are recording a Coded UI test on a login page, the “Auto Complete passwords” popup may have appeared.

When you playback actions on the login page, this dialog will *not* come up. So the action that you performed will cause an error. To improve test resilience, the Record & Playback engine automatically marks this action as “Continue on Error”.  When the test is being played back and the error occurs, the engine will continue on to the next action. Specific actions (IE popups, implicit Hovers) when performed by the user are tagged by the engine as “Continue on Error”.

In Coded UI Test, you have the flexibility to turn on “Continue on Error” for any action. This is done by the following code snippet.

Playback.PlaybackSettings.ContinueOnError = true;

Remember to turn this flag to false after the section where there are potential failures.  Otherwise the engine will ignore all errors and your test will return incorrect results. By default this flag is set to false in Coded UI Test.

2. Match Exact hierarchy

The Record and Playback Engine uses a hierarchical structure to locate controls.  The Yes button in the dialog above is identified as

To locate YesButton during playback, the engine will search for AutoCompletePasswordWindow, then for YesWindow in it, and finally YesButton in YesWindow.

Sometimes application may change so that the hierarchy gets modified. Record and Playback engine attempts to locate the control, in such cases, by skipping the intermediate elements in the hierarchy. Thus in the example, if it is unable to locate YesWindow, it will search for YesButton directly under AutoCompletePasswordWindow and all its children.  This behavior can be controlled by the “Match Exact Hierarchy” setting. This is done by the following code snippet.

Playback.PlaybackSettings.MatchExactHierarchy = true;

If the flag is set to true, every control is the hierarchy will be searched in order. Any failure will result in failure to locate the control.

By default, this flag is set to false. Note that this may sometimes lead to false positives i.e an unintended control may be identified. In such cases, you will have to change this setting.

3. Search in minimized windows

The Record and Playback Engine searches for controls even inside minimized windows. The minimized window is restored and actions performed on the control inside it.

You have the ability to turn this feature off i.e. prevent the engine from looking at minimized windows.  This is done by the following code snippet.

Playback.PlaybackSettings.SearchInMinimizedWindows = false;

4. Smart Match

The Record and Playback Engine identifies controls on the user interface by its search properties.  e.g:- The standard calculator window is identified as below.

SearchProperties[WinProperties.Window.ControlType] = ControlType.Window.Name;

SearchProperties[WinProperties.Window.Name] = "Calculator";
SearchProperties[WinProperties.Window.ClassName] = "CalcFrame";

Some of the search properties may change over time. Record and Playback engine uses a Smart Match algorithm to identify controls if they cannot be located using the exact properties. The smart match algorithm uses heuristics and attempts to locate the control using variations on the search properties.

You can control when Smart Match should be applied. By default Smart Match is applied for Top Level Windows and all controls. You can turn off Smart match using the following code snippet.

Playback.PlaybackSettings.SmartMatchOptions = SmartMatchOptions.None;

The default settings (Smart match top level windows and controls) is optimal for a resilient coded UI test. But sometimes it may lead to false positives and then you will have to modify this setting.

5. Wait For Ready Level

The Record and Playback engine ensures that each UI control is ready to be acted upon before performing any action.  The smart “Wait For Ready”  algorithm significantly improves the resilience of your Coded UI Test.  Now you don’t need to add the annoying Sleep statements whenever a UI Control is busy and not ready to receive input. By default, the engine checks the UI Thread (foreground thread) to determine if a control is ready.

You can turn off Wait For Ready completely by the following code snippet.

Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.Disabled;

Alternately you may want to wait for all threads including background threads. This may be required if the application is making asynchronous calls that results in UI changes.  You can do this by the following code snippet.

Playback.PlaybackSettings.WaitForReadyLevel = WaitForReadyLevel.AllThreads;

NOTE that this setting will slow down the engine and should be carefully used.

6.  Set Property Verification

After setting the property of any UI control, the record and playback engine performs a verification pass to ensure that the set succeeded and the UI control now has the value assigned to it.

e.g:-  You type the text “MSFT” in the search edit box in Bing. After playing back this action, the engine reads the text property of the search edit box to ensure that this set has succeeded.

In certain applications, the set property may trigger off some business logic which then changes that property itself. In such cases, the verification fails. If your application exhibits such a behavior you can turn of the set property verification using the following code snippet.

Playback.PlaybackSettings.SkipSetPropertyVerification = true;

7.  Fail Fast

The Record and playback engine’s search algorithm has a fail fast logic. i.e. If the engine is reasonably confident that the search will not succeed, it will return with a failure. This ensures that you are not waiting for a non-existent control for 2 mins (the default search timeout).

Sometimes you may want to disable fail fast. Typically, you do this along with increasing your search timeout. You are confidant that the UI control will become available within a specified duration  (which is longer than the search timeout) and you want the engine to wait for that duration.  You can do this by using the following code snippet.

Playback.PlaybackSettings.ShouldSearchFailFast = false;

8. Misc

Other playback settings which can be modified include

a.  Delay Between Actions: How long should the engine wait before performing two successive actions? By default, 100 ms.

b.  Search timeout – How long should the engine attempt to locate a control? By default, 2 mins.

c.  Wait for Ready Timeout – How long should the engine continue waiting for a control to be ready?  By Default, 60s

d. Think Time multiplier – For slow machines, assign a high multiple so that the recorded think time is scaled up to handle slower application responses.

e. Send Keys As ScanCode - By default, the engine sends keys to the application under test as Unicode. But in certain cases, it may need to send keys as scancode. Sending Unicode to such applications will result in failure

f. Encryption Key Location – When recording password fields, the record and playback engine uses a hard-coded key for encrypting the value.  A custom encryption key can be set using the SetEncryptionKeyLocation()

g. IME Languages List – These are the languages  for which IME is enabled and IME ends composition on pressing the {ENTER} key.

NOTE: The Test runner tool in VSTT 2010 also uses the Record and Playback Engine. You can configure the playback settings described above in Test runner tool from

a. mtlm.exe.config – located in %VSINSTALLDIR%\Common7\IDE

b. Microsoft Test & Lab Manager, navigate to Lab Center –> Test Settings , open the applicable Test Settings, select Data and Diagnostics section and click on the Configure button next to Action Log and Action Recording. In the Advanced section of the dialog, some of the Playback settings described above may be configured.

The managed reference for PlaybackSettings class is here.