Protocol Test Suites (Part One)

  • Article

Protocol Test Suites (Test Suites) is a toolset that enables you to validate a protocol implementation that relies on Microsoft Open Specifications.

This blog post on Test Suites is a two-part series. In this first part, let's get an overview of Test Suites, see why Test Suites are useful, examine the nuts and bolts of a Test Suite, and walk through a sample scenario. In the second part, we will go through the essentials of a Test Suite environment, examine the Test Suite folder structure, cover the end-to-end process of configuring and running Test Suites, and get an overview of different types of reports generated after a test run.

Why Use Test Suites?

When you use Open Specifications protocols, you can extend or replace a Microsoft client or server in your implementation. An essential part of the process is to determine whether the implementation is accurate and conforms to Open Specifications documentation. This is where Test Suites come in handy! Test Suites contain a set of automated test cases that run on your implementation and validate whether the protocols are set up in accordance with the Open Specifications documentation.

Test Suites do not cover every protocol requirement and do not certify an implementation, even if all tests pass. However, each Test Suite provides users with a useful indication of whether your implementation is set up correctly or not. Reports and log files generated after a Test Suite has been run provide useful debug information to help you identify what failed. Source code is provided, allowing you to extend and integrate the Test Suites as needed for your implementation.

Protocol Implementation Scenario

The following diagram illustrates the most common scenario for protocol implementation and shows how Test Suites fit into the greater scheme of things.

Figure 1: Protocol implementation scenario

Note: Test Suites focus only on the Man-in-the-Middle and Server implementation scenarios.

By acting as a synthetic client, Test Suites validate the server-side implementation of the protocols. The server side of the Test Suite environment is called a System Under Test (SUT). The SUT can either be a Microsoft implementation (an Exchange or a SharePoint server) of the protocol or a third-party implementation.

The Test Suite client verifies that the server behaves in a way that is compliant with normative protocol requirements, as described in the technical specification associated with the Test Suite. Test Suites are not designed to run multi-protocol user scenarios, but are intended to validate certain operations documented in a technical specification for a protocol. The Test Suites are functional tests that verify the compatibility of the SUT with protocol implementation.

Test Suite Design and Components

Before we delve into the high-level design of a Test Suite, let's get a little background on how Test Suites are developed.

Every protocol technical document contains normative statements, which identify information required for interoperability. Normative statements contain keywords such as MUST, MAY, and SHOULD. Normative statements are interpreted as requirements. The requirements are evaluated to determine if they can become test cases, and if they can be, then the source code for the test cases are written. Test cases are then logically grouped together to form scenarios.

Figure 2: Test Suite development

Let's take a look at how scenarios and test cases fold into the high-level design of a Test Suite.

Figure 3: High-level design of a Test Suite

The nuts and bolts of a Test Suite are mainly comprised of test cases and adapters.

  • Test Case: A test case is a group of programs and scripts that validate how your implementation conforms to the technical specification. A series of test cases is called a scenario.
  • Adapters: An adapter is an interface between the Test Suite and the SUT. Each Test Suite has two adapters, a Protocol Adapter and an SUT Control Adapter.

    • Protocol Adapter: The Protocol Adapter is used to initialize the connection to the SUT, generate request protocol messages, and consume protocol response messages. This type of adapter also creates and maintains the connection with the SUT.

    • SUT Control Adapters: The SUT Control Adapter is used to query or configure the SUT. An SUT Control Adapter is typically used when a test case has to control the server outside of the protocol. For example, in the Exchange Test Suites, when running a test case from the Store Object Protocol (defined in MS-OXCSTOR), the process of creating and deleting a mailbox is managed by the SUT Control Adapter. In the SharePoint Test Suites, the setup and tear down (creating or removing subfolders or email items on the SUT) are managed through the SUT Control Adapter.

      The SUT Control Adapter is primarily designed to work with the Microsoft implementation of the SUT. However, if you are replacing a Microsoft server with your own, you will be likely replacing the SUT adapter with the one that works with your server.

      The SUT control adapter can use one of the following types, by simply modifying the Test Suite Configuration file:

    • PowerShell script adapter: This adapter maps interface methods to PowerShell scripts.

    • Managed code adapter: This adapter implements interface methods in managed code (C#).

    • Interactive adapter: This adapter can be used when the server requires manual configuration.

      Note: More information on modifying the test suite configuration file is available in the Test Suite Deployment Guide within a test suite folder where the Test Suite has been installed. For more information on the Test Suite folder and its contents, see Protocol Test Suites (Part Two).

Example: Creating an Attachment

Here is an example to illustrate how a test suite simulates a protocol in a real-world implementation.
 
Whenever you send an email with an attachment using Microsoft Outlook, behind the scenes, Outlook sends out the RopCreateAttachmentRequest request (defined in MS-OXCMSG) to the Exchange server, which in turn sends the RopCreateAttachmentResponse response back to Outlook.

When you run the MS-OXCMSG Test Suite against your implementation, the Test Suite generates the protocol traffic by acting as a synthetic client. The Test Suite sends a similar RopCreateAttachmentRequest request to the SUT, and will expect the RopCreateAttachmentResponse back from the server. If your implementation complies with the normative requirements as defined in MS-OXCMSG, the SUT will send the RopCreateAttachmentResponse response back to Outlook. 

Figure 4: Create an attachment

That’s all for now! In Part Two, we will cover the prerequisites of setting up a test environment, examine the structure of the Test Suite folder, and explain configuration, and debugging and results analysis.