How to customize test initialization and logging

Junfeng Dai has written an article about customizing Spec Explorer for using NUnit as the test execution engine instead of VSTT a few month ago. The same general approach can be used for simpler scenarios, for example to add custom test initialization code or test logging, which I would like to explain here.

When Spec Explorer generates test code, this code is placed in a class which derives from a base class found in the Spec Explorer runtime. You can simply insert a ‘middle man’ between this class and the generated test code class, overriding some methods.

Suppose you have created a Spec Explorer project based on the ‘static modeling solution’ (the same applies for any other Spec Explorer project), and you have generated test code.  You will see that the test code looks as below:

    1:  namespace SpecExplorer1.TestSuite {

    2:      using System;

    3:      using System.Collections.Generic;

    4:      using System.Text;

    5:      using System.Reflection;

    6:      using Microsoft.SpecExplorer.Runtime.Testing;

    7:      

    8:      

    9:      [System.CodeDom.Compiler.GeneratedCodeAttribute("Spec Explorer", "3.0.2168.0")]

   10:      [Microsoft.VisualStudio.TestTools.UnitTesting.TestClassAttribute()]

   11:      public partial class AccumulatorTestSuite : VsTestClassBase {

   12:          

   13:          public AccumulatorTestSuite() { 

   14:             ...

   15:          }

The class VsTestClassBase in line 11 is what we would like to substitute. However, in contrast to the full replacement scenario, we want to still use most of its functionality, and continue to run our tests under VSTT.

So we add a class to the test suite project as below, where we override a couple of methods from VsTestClassBase:

 namespace SpecExplorer1.TestSuite
{
    public class MyTestClassBase : VsTestClassBase
    {
        public override void InitializeTestManager()
        {
            base.InitializeTestManager();
            Console.WriteLine("My test initialization executes now");
        }

        public override void CleanupTestManager()
        {
            Console.WriteLine("My test cleanup executes now");
            base.CleanupTestManager();
        }

        public override void Comment(string description)
        {
            Console.WriteLine("My test logging executes now: {0}", description);
        }

        public override void Checkpoint(string description)
        {
             Console.WriteLine("My requirement logging executes now: {0}", description);
        }

    }
}

In the methods InitializeTestManager and CleanupTestManager, we add custom test preparation and cleanup code. Note that it is imperative to call the base class here (unless you want to install your own test manager, which is usually not required).

Finally, we just need to tell Spec Explorer to use our new test class base when it generates test code, instead of the standard one. This is achieved by setting the switch TestClassBase in the Cord configuration:

    1:  /// Contains actions of the model, bounds, and switches.

    2:  config Main 

    3:  {

    4:      /// The two implementation actions that will be modeled and tested

    5:      action abstract static void Accumulator.Add(int x);

    6:      action abstract static int Accumulator.ReadAndReset();

    7:   

    8:      switch StepBound = 128;

    9:      switch PathDepthBound = 128;

   10:      switch TestClassBase = "MyTestClassBase";

   11:      switch GeneratedTestPath = "..\\TestSuite";

   12:      switch GeneratedTestNamespace = "SpecExplorer1.TestSuite";

   13:      switch TestEnabled = false;

   14:      switch ForExploration = false;

   15:  }

Regenerating test code and executing it finally yields in the following test log:

 My test initialization executes now
My test logging executes now: reaching state 'S0'
My test logging executes now: executing step 'call Add(2)'
My requirement logging executes now: Add rule captured
My test logging executes now: reaching state 'S1'
My test logging executes now: checking step 'return Add'
My test logging executes now: reaching state 'S8'
My test logging executes now: executing step 'call Add(2)'
My requirement logging executes now: Add rule captured
My test logging executes now: reaching state 'S12'
My test logging executes now: checking step 'return Add'
My test logging executes now: reaching state 'S16'
My test logging executes now: executing step 'call ReadAndReset()'
My test logging executes now: reaching state 'S20'
My test logging executes now: checking step 'return ReadAndReset/4'
My test logging executes now: reaching state 'S24'
My test cleanup executes now