How to Boost Performance of the ESB Routing and Transform Services – Part 1

  • Articolo

Introduction

Recently I had chance to work with a couple of customers that make extensive use the ESB Toolkit (specifically, V 2.0 in this case) in their BizTalk solutions, and to exchange ideas with some of you regarding the alleged performance problems affecting Dynamic Send Ports. In the last few weeks I received comments like the following:

  • We had to revert from dynamic ports to static ports when we deployed the solution to production due to performance problems.
  • Your suspicion was correct. It was the performance of dynamic send ports that was the cause of the poor performance.
  • etc.

Therefore, I decided to take the plunge, put on my (Lego) Indiana Jones hat, put my favorite gadgets (Reflector Pro, Visual Studio Profiler, SQL Server Profiler, etc.) in my backpack and enter the forest of the ESB Toolkit to make my own investigations.  
This article is quite long, so I decided to break it down in 3 parts in the following way:

  • Part 1: Problem Statement and Solution
  • Part 2: Scenarios and Test Use Cases
  • Part 3: Performance Tests Results

Dynamic Send Ports

The use of Dynamic Send Ports confers great flexibility to any BizTalk applications, not only to those using the ESB Toolkit, because they provide the ability to dynamically choose within a Receive Location or an Orchestration, the proper Adapter to use, set its context properties, and finally specify the target URL. However, they also present the following challenges:

  • When creating a Dynamic Send Port, the BizTalk Administration Console does not provide the ability to define Transport parameters like the Retry Count and Retry Interval or the possibility to select a specific Send Handler. At runtime you can dynamically set the value of the BTS.RetryCount and BTS.RetryInterval context properties by code within an Orchestration or a custom Pipeline Component to specify a different value, respectively, for the Retry Count and Retry Interval parameters, but there's no way to select a specific Send Handler. Therefore, at run-time a Dynamic Send Port runs on the default Host configured for the adapter that is being used. As a consequence, Dynamic Send Ports sharing the same adapter are forced to execute within the same host. For more information on this topic, you can review the following blog posts:

  • The Adapter context properties must be carefully set within a Receive Location or an Orchestration to properly configure and drive the run-time behavior of Dynamic Send Ports. For example, when using the WCF-Custom Adapter, you need to properly define properties like WCF.Action, WCF.BindingType, WCF.BindingConfiguration, etc. as shown in the code snippet below:

     MessageOut=MessageIn;MessageOut(WCF.BindingType)="customBinding";MessageOut(WCF.Action)="https://tempuri.org/IReceiveMessage/ReceiveMessage";MessageOut(WCF.BindingConfiguration)=@"<binding name=""customBinding""><binaryMessageEncoding /><tcpTransport /></binding>";DynamicSendPort(Microsoft.XLANGs.BaseTypes.Address)="net.tcp://localhost:8001/customNetTcp";DynamicSendPort(Microsoft.XLANGs.BaseTypes.TransportType)="WCF-Custom";

  • For more information on this subject, you can review the BizTalk online documentation:

    • "Configuring Dynamic Send Ports Using WCF Adapters Context Properties" on MSDN.
    • "Configuring Dynamic Send Ports" on MSDN.

  • Failing to adequately define adapter context properties can easily lead to performance problems or unexpected behaviors.
    For more information on this topic, please review the following articles:

    • "Performance Tip when using WCF-Custom with Dynamic Send Ports and Custom Bindings" on the Microsoft BizTalk Server Blog.
    • "Configure EnableTransaction and IsolationLevel property in Business Rules for Dynamic Send Port" on Vishal Mody's Blog.

  • Sometimes Adapters have a different behavior depending on whether they are used in a Static Send Port or a Dynamic Send Port. Since Dynamic Ports are dynamically configured on every call, any preliminary actions undertaken by the Adapter being used is re-executed every time the port handles a new message. Therefore, when using a WCF Adapter (e.g. WCF-Custom) on a Dynamic Send Port, the channel stack used to invoke the target service is re-created at each call and this operation is quite expensive in terms of performance. Moreover, when using the WCF-SQL Adapter or the WCF-Custom Adapter + SqlBinding in a Dynamic Send Port to invoke a stored procedure on a custom DB, the execution of the stored procedure is always preceded by a statement that retrieves metadata about the stored procedure's parameters. I'll provide you evidence of this behavior in the reminder of this article. This has a heavy impact on overall performance. I didn't have a chance to run tests using all the adapters, so I can’t confirm the following statements for 100% of the adapters, but the general observation is that if the Adapter being used by a Dynamic Send Port needs to perform some warm-up actions like creating a proxy, retrieving metadata or configuration data, establishing a connection to the target remote system, etc. these actions get re-executed upon each message transmission. Clearly, when an adapter does exhibit this behavior it seriously impairs overall performance. The large amount of overhead due to the use of Dynamic Send Ports in place of Static Send Ports, depends on the weight and cost of warm-up operations performed by the Adapter being used: if the Adapter in question is the WCF-SQL, the cost in terms of performance can be quite heavy, as highlighted above, whereas the overhead should be minimal when using the FILE Adapter . That’s why some of you have thought to replace Dynamic Send Ports with Static Send Ports using different techniques. I’ll show you how to accomplish this task using a custom Routing Service in the reminder of this article.
    For more information on this topic, you can review the following blog posts: :

  • Each individual Dynamic Send Port has unique Activation Subscriptions for each Adapter installed in the BizTalk environment, as shown in the following figure. To a certain extent, this can increase the cost of subscription matching at runtime.

Subscriptions

ESB Routing Service and Dynamic Send Ports

The ESB Toolkit encourages and promotes the use of Dynamic Send Ports. When you create an Itinerary and in particular when you configure an Off-Ramp Service, the designer allows you to select one of the Dynamic Send Ports available within the selected BizTalk application, while Static Send Ports are not supported. In other words, by default, you cannot use the Microsoft.Practices.ESB.Services.Routing service provided out of the box by the ESB Toolkit with Static Send Ports. However, there’s an easy and obvious workaround for this problem. To use a Static Send Port in place of a Dynamic Send Port you can proceed as follows:

  • Before designing a new Itinerary, you can create a Dynamic Send Port using the BizTalk Administration Console or a script.
  • As part of the Port definition, you specify a Filter Expression as shown in the following figure:

DynamicSendPort

  • Then in your Itinerary, you select and use the Dynamic Send Port as placeholder when defining an Off-Ramp Service, as highlighted in red in the following figure.

Off-Ramp

  • Using the BizTalk Administration Console or a script, you unenlist the Dynamic Send Port. This way you prevent the Dynamic Send Port from receiving and handling any messages at runtime as the final objective of this pattern is using a Static Send Port in its place.
  • Finally, you create a Static Send Port with the same filter expression as the Dynamic Send Port.  You cannot select a Static Send Port in an itinerary, but by setting the ServiceName property in the Filter Expression to the same as the Dynamic Send Port, the subscription is satisfied and the port will receive the message.

For more information on this topic, you can also review the following blog posts:

A drawback of this technique is that it requires the creation of a placeholder Dynamic Send Port as well as a corresponding Static Send Port for each remote endpoint. So I asked myself the following question: is there a better way to use Static Ports with the ESB Toolkit? Can I avoid creating several Static Ports, one for each target endpoint or system? The answer is yes and the effort to accomplish this goal is minimal.

I started noting that in the majority of BizTalk applications, Dynamic Send Ports are exploited just to change the target URL, that is the BTS.OutboundTransportLocation context property, within a pipeline component in a Receive Location or inside an Orchestration, while the Adapter does not vary. This task can be easily accomplished using the Routing Service (Messaging or Orchestration), and one of the Resolvers provided out-of-the-box by the ESB Toolkit (UDDI, BRE, STATIC, XPATH, etc.). So I asked myself the following question: can I define a Static Send Port and use it to exchange data with multiple, distinct target systems just by changing the context properties, like the aforementioned BTS.OutboundTransportLocation, before transmitting outbound messages?

When you create a Static Send Port you have to specify, in a declarative way at configuration time, the target URL and Adapter-specific properties (for example, the Action property, when defining a WCF Send Port). At runtime, if you try to dynamically change, within a Receive Location or an Orchestration, the value of the corresponding context properties like the BTS.OutboundTransportLocation or the WCF.Action, the Static Send Port will ignore these values and continue to use the data statically specified as part of the Port definition. However, this rule applies only if you try to change general and Adapter-specific context properties before publishing the message to the MessageBox. In fact, if you try to change the value of context properties like BTS.OutboundTransportLocation and WCF.Action within a pipeline component in the transmit pipeline used by the Static Send Port, the Adapter Send Handler will ignore the values configured on the Static Port and will use those specified by the pipeline component. 

 Quotes_Icon Note This technique requires that you follow  some rules: for example, when using a WCF Static Send Port, make sure to delete the content of the Action field on the WCF-* Adapter configuration. This condition is necessary to allow the send pipeline to dynamically define a value for the WCF.Action context property.

 

To confirm my theory, I made a sample in which a custom pipeline component dynamically changes the target URL (from the URL defined during configuration) on a FILE Send Port by assigning a new value to the BTS.OutboundTransportLocation context property within the send pipeline used by the Port.

If you look at the samples shipped with the ESB Toolkit or published by bloggers on the Internet, that make use of the Routing Messaging Service, you can easily note that the value of the Container property of this latter service is always equal to OnRamp.receiveInbound, as shown in the picture below.

SimpleRoutingSlip

This means that the service in question will be executed within a Receive Location configured to use one of the receive pipelines provided by the ESB Toolkit or a custom receive pipeline that contains the ESB Dispatcher component. When using the Routing or Transform Orchestration service provided out-of-the-box by the ESB Toolkit or a custom Orchestration Service, you can retrieve routing data using a Resolver defined on the service itself or on the Off-Ramp Extender, as shown in the following figure.

OrchestrationRoutingSlip

In any case when you advance the Itinerary to the next Itinerary Step, the Resolver will be executed and the corresponding data used to initialize routing-related context properties (BTS.OutboundTransportLocation and BTS.OutboundTransportType). In both cases, routing-related context properties will be set before posting the message to the MessageBox. Now, as we noted before, this pattern permits routing the message to a Dynamic Send Port and initializes the target URL, but it does not allow overriding the URL statically defined on a Static Send Port. Therefore, I tried to use a different approach and I moved both the Transform Service and the  Routing Service on the Off-Ramp Send Port by selecting OffRamp.sendTransmit as the value for their Container property (see the picture below):

StaticSendPortRoutingSlip

As described in the first part of this article, I used a placeholder Dynamic Send Port to define the Off-Ramp Service within my Itinerary, but then I replaced this with the real one-way FILE Static Send Port that uses the same filter expression; the STATIC Resolver defined on the Routing Service (see the figure below) has been configured to write the incoming message to a folder other than that used as a target by the above FILE Send Port:

STATICResolver

To test my Itinerary I built the use case represented by the following picture:

RoutingSlipFlow

Message Flow:

  1. A One-Way FILE Receive Location receives a new xml document from a Client Application.
  2. The ESB Itinerary Selector pipeline component running within the ItinerarySelectReceivePassthrough pipeline retrieves the Itinerary from the EsbItineraryDb or from the in-process cache and copies it in the ItineraryHeader context property. Then the ESB Dispatcher pipeline component advances the Itinerary to the next step.
  3. The Message Agent submits the incoming message to the MessageBox (BizTalkMsgBoxDb).
  4. The message is retrieved by a One-Way FILE Static Send Port.
  5. The ESB Dispatcher pipeline component running within the ItinerarySendPassthrough pipeline retrieves the Itinerary from the ItineraryHeader context property and executes in sequence the Transform Service and Routing Service. In particular, the Routing Service changes the value of the OutboundTransportLocation and Action context properties using data defined in the corresponding STATIC Resolver. Finally,  the ESB Dispatcher pipeline component advances the Itinerary to the next step.
  6. The Send Handler of the FILE Adapter finally writes the message to the output folder indicated in the Itinerary.

Everything was in place, I was ready to go, I gave it a try, but it didn’t work as expected : in fact, the message was copied in the folder referenced by the Static Send Port rather than in the folder indicated by the STATIC Router used by the Routing Service. Therefore, I decided to start using my favorite tool ( .NET Reflector PRO) to investigate. Analyzing and debugging through the code, I finally found the culprit line of code: the Execute method exposed by the Microsoft.Practices.ESB.Itinerary.Services.RoutingService class contains the following code:

 

 public IBaseMessage Execute(IPipelineContext context, IBaseMessage msg, string resolverString, IItineraryStep step){                // only end point resolution if this is a Receive Inbound     if (step.MessageDirection == ItineraryMessageDirection.ReceiveInbound ||         step.MessageDirection == ItineraryMessageDirection.SendInbound)    {        return ExecuteRoute(context, msg, resolverString);    }    return msg;}

 

In practice, when running within a send pipeline, the Microsoft.Practices.ESB.Itinerary.Services.RoutingService provided out-of-the-box by the ESB Toolkit just returns the incoming message without performing any . Therefore I decided to create my own version of this component: using .NET Reflector PRO I disassembled and copied the code of the original class and I created a new custom class called Microsoft.BizTalk.CAT.ESB.Itinerary.Services.RoutingService where I removed the if condition from the Execute method as shown in the picture below.

 public IBaseMessage Execute(IPipelineContext context, IBaseMessage message, string resolverString, IItineraryStep step){    return ExecuteRoute(context, message, resolverString);}

 

Then I installed my component in the GAC and registered it as a service  in the ESB Toolkit configuration file (esb.config). Finally I replaced the original RoutingService in my Itinerary with the new custom service. At this point I crossed my fingers and ran a new test: this time the use case behaved as expected and the FILE Adapter wrote the message in the path specified by the STATIC Resolver within the Itinerary.
 

 Quotes_Icon Note This technique consisting in changing the value of the OutboundTransportLocation property within a Static Send Port  is not guaranteed to work with all the BizTalk Adapters: for example, I tried to use my custom Routing Service with an MQSeries Static Send Port, but it didn't work as expected. In other words, the MQSeries Adapter ignores the value set by the Routing Service within the ItinerarySendPassthrough pipeline and continues to use the URL statically defined in the Queue Definition property inside the Adapter configuration.

 

The next step was to review the code of the default TransformService class to see if it could be optimized for best performance. This brings us to the next point.

Transform Service

Once again, I used .NET Reflector to disassemble and inspect the code of the TransformService class contained in the Microsoft.Practices.ESB.Itinerary.Services assembly. For you convenience, I reported below just the code of the method TransformStream that is responsible for applying a transformation map to the stream containing the message.

 private static Stream TransformStream(Stream stream, string mapName, bool validate, ref string messageType){                                 Type mapType = Type.GetType(mapName);    if (null == mapType)        throw new TransformationServiceException(Properties.Resources.InvalidMapType, mapName);    TransformMetaData metaData = TransformMetaData.For(mapType);        SchemaMetadata schema = metaData.SourceSchemas[0];    String sourceMap = schema.SchemaName;                       SchemaMetadata targetSchema = metaData.TargetSchemas[0];    if (validate)    {        if (string.Compare(messageType, sourceMap, false, CultureInfo.CurrentCulture) != 0)            throw new TransformationServiceException(Properties.Resources.SourceDocumentMismatch, messageType, sourceMap);    }    messageType = targetSchema.SchemaName;    XPathDocument doc = new XPathDocument(stream);    XslTransform transform = metaData.Transform;    Stream output = new MemoryStream();    transform.Transform(doc, metaData.ArgumentList, output);    output.Flush();    output.Seek(0, SeekOrigin.Begin);    return output;}

 

As you can easily note, the method starts using the fully qualified name of the map contained in the mapName parameter to retrieve the type of the map, then it uses this object to retrieve the map metadata and finally applies the map to the message contained in the stream parameter using a XslTransform object. Reviewing the code above I identified 2 potential optimizations:

  • Since retrieving the map type using the Type.GetType call is quite expensive from a performance perspective as any other Reflection call, and since the same map will be reused several times over time, this information can be cached in-process within a static structure, for instance a Dictionary.
  • The XslTransform object is an inefficient way to apply an XSLT to a document.

As you probably know, the BizTalk Runtime still makes an extensive use of the System.Xml.Xsl.XslTransform: for instance, when you create and build a BizTalk project, a separate .NET class is generated for each transformation map. Each of these classes inherits from the Microsoft.XLANGs.BaseTypes.TransformBase class.

When BizTalk Server 2004 was built, the  XslTransform was the only class provided by the Microsoft .NET Framework 1.1 to apply an XSLT to an inbound XML document. When the Microsoft .NET Framework version 2.0. was released, the  XslTransform was declared obsolete and  thus deprecated. As clearly stated on MSDN, the System.Xml.Xsl.XslCompiledTransform should be used instead. This class is used to compile and execute XSLT transformations. In most cases, the XslCompiledTransform class significantly outperforms the XslTransform class in terms of time need to execute the same XSLT against the same inbound XML document. The article Migrating From the XslTransform Class on MSDN reports as follows:

“The XslCompiledTransform class includes many performance improvements. The new XSLT processor compiles the XSLT style sheet down to a common intermediate format, similar to what the common language runtime (CLR) does for other programming languages. Once the style sheet is compiled, it can be cached and reused.”

The caveat is that because the XSLT is compiled to MSIL, the first time the transform is run there is a performance hit, but subsequent executions are much faster. To avoid paying the extra cost of initial compilation every time a map is executed, this latter could be cached in a static structure (e.g. Dictionary). I’ll show you how to implement this pattern in the second part of the article. For a detailed look at the performance differences between the  XslTransform and XslCompiledTransform classes (plus comparisons with other XSLT processors) have a look at the following posts.

Although the overall performance of the XslCompiledTransform class is better than the XslTransform class, the Load method of the XslCompiledTransform class might perform more slowly than the Load method of the XslTransform class the first time it is called on a transformation. This is because the XSLT file must be compiled before it is loaded. However, if you cache an XslCompiledTransform object for subsequent calls, its Transform method is incredibly faster than the equivalent Transform method of the XslTransform class. Therefore, from a performance perspective:

  • The XslTransform class is the best choice in a "Load once, Transform once" scenario as it doesn't require the initial map-compilation.
  • The XslCompiledTransform class is the best choice in a "Load once, Cache and Transform many times" scenario as it implies the initial cost for the map-compilation, but then this overhead is highly compensated by the fact that subsequent calls are much faster.

As BizTalk is a server application (or, if you prefer an application server), the second scenario is more likely than the first.  The only way to take advantage of this class (given that BizTalk does not currently make use of the XslCompiledTransform class) is to write custom components. If this seems a little strange to you, remember that all BizTalk versions since BizTalk Server 2004 have inherited that core engine, based on .NET Framework 1.1.  Since the XslCompiledTransform class wasn’t added until .NET Framework 2.0, it wasn’t leveraged in that version of BizTalk. While I’m currently working with the BizTalk Development Team to see how best to take advantage of this class in the next version of BizTalk, some months ago I decided to create a helper class that, exploiting the capabilities provided by the XslCompiledTransform class, can significantly boost the performance of transformations in BizTalk application. For more information, you can read my previous posts on this subject:

  • “How To Boost Message Transformations Using the XslCompiledTransform class” on my blog.
  • “How To Boost Message Transformations Using the XslCompiledTransform class Extended” on my blog.

Taking into account all these considerations, I decided to create a new, faster version of the Transform Service. Once again I started from the code of the original TransformService class, but this time I radically changed its code to cache map metadata and exploit the capabilities provided by the XslCompiledTransform class. For your convenience, I reported below the code of the custom TransformService class and a customized version of my XslCompiledTransformHelper class. In particular, the following 2 optimizations have been implemented in the TransformStream method:

  • The schema name of both the source and target documents of the map is cached in static Dictionary.
  • The use of the XslTransform class has been replaced with the use of an XslCompiledTransformHelper helper component that internally compiles, caches and applies maps to inbound messages using an instance of the XslCompiledTransform class.

TransformService  Class

 #region Copyright//-------------------------------------------------// Author:  Paolo Salvatori// Email:   paolos@microsoft.com// History: 2010-06-15 Created//-------------------------------------------------#endregion#region Using Directivesusing System;using System.IO;using System.Reflection;using System.Xml.XPath;using System.Xml.Xsl;using System.Configuration;using System.Collections.Generic;using System.ComponentModel;using System.Globalization;using Microsoft.BizTalk.Component.Interop;using Microsoft.BizTalk.Message.Interop;using Microsoft.BizTalk.Streaming;using Microsoft.Practices.ESB.Adapter;using Microsoft.Practices.ESB.Itinerary;using Microsoft.Practices.ESB.Exception.Management;using Microsoft.Practices.ESB.GlobalPropertyContext;using Microsoft.Practices.ESB.Resolver;using Microsoft.Practices.ESB.Utilities;using Microsoft.XLANGs.RuntimeTypes;using Microsoft.BizTalk.CAT.ESB.Itinerary.Services.Properties;#endregionnamespace Microsoft.BizTalk.CAT.ESB.Itinerary.Services{    /// <summary>    /// Service responsible for setting up the message for transformation and executing the transformation.    /// </summary>    public class TransformationService : IMessagingService    {        #region Private Constants        private const string ValidateSourceKey = "ValidateSource";        private const string BufferSize = "bufferSize";        private const string ThresholdSize = "thresholdSize";        #endregion        #region Private Static Fields        private static Dictionary<string, TypeInfo> typeInfoCollection = new Dictionary<string, TypeInfo>();        private static int bufferSize = XslCompiledTransformHelper.DefaultBufferSize;        private static int thresholdSize = XslCompiledTransformHelper.DefaultThresholdSize;        #endregion        #region Static Constructor        static TransformationService()        {            try            {                string setting = null;                setting = ConfigurationManager.AppSettings[BufferSize];                if (!string.IsNullOrEmpty(setting))                {                    if (!int.TryParse(setting, out bufferSize))                    {                        bufferSize = XslCompiledTransformHelper.DefaultBufferSize;                    }                }                setting = ConfigurationManager.AppSettings[ThresholdSize];                if (!string.IsNullOrEmpty(setting))                {                    if (!int.TryParse(setting, out thresholdSize))                    {                        thresholdSize = XslCompiledTransformHelper.DefaultThresholdSize;                    }                }            }            catch (Exception)            {            }        }        #endregion        #region IMessagingService Members        /// <summary>        /// Name of the service.        /// </summary>        /// <remarks>        /// This name must match the configured name in Itinerary services configuration.        /// </remarks>        [Browsable(true)]        [Description("Name of the service.  This name must match the configured name in Itinerary services configuration.")]        [ReadOnly(true)]        public string Name        {            get             {                return Resources.TransformServiceName;             }        }        /// <summary>        /// The routing service does not support disassemble and execution of multiple resolvers.        /// </summary>        public bool SupportsDisassemble        {            get             {                 return false;             }        }        /// <summary>        /// Determines if the step should be advanced.        /// </summary>        /// <param name="step">Current step information from the itinerary.</param>        /// <param name="msg">Current message in pipeline.</param>        /// <returns>True</returns>        public bool ShouldAdvanceStep(IItineraryStep step, IBaseMessage msg)        {            return true;        }        /// <summary>        /// Execute a transformation of a message within the lifecyle of an itinerary.        /// </summary>        /// <param name="step">Current step in the itinerary</param>        /// <param name="context">Pipeline context information</param>        /// <param name="message">Message coming through the pipeline</param>        /// <param name=Resources.ResolverString>Resolver string that needs to be resolved by the appropriate resolver.</param>        /// <returns>The transformed message.</returns>        public IBaseMessage Execute(IPipelineContext context, IBaseMessage message, string resolverString, IItineraryStep step)        {            if (context == null)            {                throw new ArgumentNullException(Resources.ContextParameter, Resources.ContextCannotBeNull);            }            if (message == null)            {                throw new ArgumentNullException(Resources.MessageParameter, Resources.MessageCannotBeNull);            }            if (string.IsNullOrEmpty(resolverString))            {                throw new ArgumentException(Resources.ArgumentStringRequired, Resources.ResolverString);            }            try            {                bool validateSource = false;                if (step != null && step.PropertyBag.ContainsKey(ValidateSourceKey))                {                    validateSource = string.Compare(step.PropertyBag[ValidateSourceKey], "true", true, CultureInfo.CurrentCulture) == 0;                }                return ExecuteTransform(context, message, resolverString, validateSource);            }            catch (Exception ex)            {                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);                throw;            }                 }        /// <summary>        /// Execute a transformation of a message.  This operation is generally used outside of an itinerary.        /// </summary>        /// <param name="context">Pipeline context information</param>        /// <param name="message">Message coming through the pipeline</param>        /// <param name="mapNameOrResolverString">Map name including assembly details or resolver string to get the map name.</param>        /// <param name="validateSource">Determine if source message type is validated against map source message type.</param>        /// <returns>The transformed message</returns>        public IBaseMessage ExecuteTransform(IPipelineContext context, IBaseMessage message, string mapNameOrResolverString, bool validateSource)        {            if (context == null)            {                throw new ArgumentNullException(Resources.ContextParameter, Resources.ContextCannotBeNull);            }            if (message == null)            {                throw new ArgumentNullException(Resources.MessageParameter, Resources.MessageCannotBeNull);            }            if (string.IsNullOrEmpty(mapNameOrResolverString))            {                throw new TransformationServiceException(Resources.MapNameRequired);            }            try            {                string mapName = string.Empty;                ResolverInfo info = ResolverMgr.GetResolverInfo(ResolutionType.Transform, mapNameOrResolverString);                if (info.Success)                {                    Dictionary<string, string> resolverDictionary = ResolverMgr.Resolve(info, message, context);                    mapName = resolverDictionary["Resolver.TransformType"];                }                else                {                    mapName = mapNameOrResolverString;                }                if (string.IsNullOrEmpty(mapName))                {                    throw new TransformationServiceException(Resources.MapNameRequired);                }                Stream stream = message.BodyPart.GetOriginalDataStream();                // If not seekable, make it so                if (!stream.CanSeek)                {                    // Create a virtual and seekable stream                    Stream virtualStream = new VirtualStream(bufferSize, thresholdSize);                    Stream seekableStream = new ReadOnlySeekableStream(stream, virtualStream, bufferSize);                    stream = seekableStream;                }                stream.Position = 0;                string messageType = message.Context.Read(BtsProperties.MessageType.Name, BtsProperties.MessageType.Namespace) as string;                                if (string.IsNullOrEmpty(messageType))                {                    messageType = MessageHelper.GetMessageType(message, context);                }                IBaseMessage outMsg = MessageHelper.CreateNewMessage(context,                                                                      message,                                                                     TransformStream(stream,                                                                                      mapName,                                                                                      validateSource,                                                                                      (int)bufferSize,                                                                                      (int)thresholdSize,                                                                                      ref messageType));                                outMsg.Context.Write(BtsProperties.SchemaStrongName.Name, BtsProperties.SchemaStrongName.Namespace, null);                outMsg.Context.Write(BtsProperties.MessageType.Name, BtsProperties.MessageType.Namespace, messageType);                MessageHelper.SetDocProperties(context, outMsg);                return outMsg;            }            catch (Exception ex)            {                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);                throw;            }        }        #endregion        #region Private Static Methods        private static Stream TransformStream(Stream stream, string mapName, bool validate, int bufferSize, int thresholdSize, ref string messageType)        {            TypeInfo typeInfo = null;            lock (typeInfoCollection)            {                if (typeInfoCollection.ContainsKey(mapName))                {                    typeInfo = typeInfoCollection[mapName];                }                else                {                    typeInfo = new TypeInfo();                    Type mapType = Type.GetType(mapName);                    if (null == mapType)                    {                        throw new TransformationServiceException(Resources.InvalidMapType, mapName);                    }                    TransformMetaData transformMetaData = TransformMetaData.For(mapType);                    if (transformMetaData.SourceSchemas.Length > 0 &&                        transformMetaData.SourceSchemas[0] != null &&                        !string.IsNullOrEmpty(transformMetaData.SourceSchemas[0].SchemaName))                    {                        typeInfo.SourceSchemaName = transformMetaData.SourceSchemas[0].SchemaName;                    }                    if (transformMetaData.TargetSchemas.Length > 0 &&                        transformMetaData.TargetSchemas[0] != null &&                        !string.IsNullOrEmpty(transformMetaData.TargetSchemas[0].SchemaName))                    {                        typeInfo.TargetSchemaName = transformMetaData.TargetSchemas[0].SchemaName;                    }                    typeInfoCollection.Add(mapName, typeInfo);                }            }                        if (validate)            {                if (string.Compare(messageType, typeInfo.SourceSchemaName, false, CultureInfo.CurrentCulture) != 0)                {                    throw new TransformationServiceException(Resources.SourceDocumentMismatch, messageType, typeInfo.SourceSchemaName);                }            }            messageType = typeInfo.TargetSchemaName;            return XslCompiledTransformHelper.Transform(stream, mapName, true, bufferSize, thresholdSize);        }                  #endregion    }    public class TypeInfo    {        #region Private Fields        private string sourceSchemaName = null;        private string targetSchemaName;        #endregion        #region Public Constructors        public TypeInfo()        {        }        public TypeInfo(string sourceSchemaName,                        string targetSchemaName)        {            this.sourceSchemaName = sourceSchemaName;            this.targetSchemaName = targetSchemaName;        }        #endregion        #region Public Properties        public string SourceSchemaName        {            get            {                return this.sourceSchemaName;            }            set            {                this.sourceSchemaName = value;            }        }        public string TargetSchemaName        {            get            {                return this.targetSchemaName;            }            set            {                this.targetSchemaName = value;            }        }        #endregion    }}

TransformService  Class

 #region Copyright//-------------------------------------------------// Author:  Paolo Salvatori// Email:   paolos@microsoft.com// History: 2010-01-26 Created//-------------------------------------------------#endregion#region Using Referencesusing System;using System.IO;using System.Text;using System.Collections.Generic;using System.Configuration;using System.Xml;using System.Xml.Xsl;using System.Xml.XPath;using System.Diagnostics;using System.Reflection;using Microsoft.XLANGs.BaseTypes;using Microsoft.XLANGs.Core;using Microsoft.BizTalk.Streaming;using Microsoft.Practices.ESB.Exception.Management;using Microsoft.BizTalk.CAT.ESB.Itinerary.Services.Properties;#endregionnamespace Microsoft.BizTalk.CAT.ESB.Itinerary.Services{    public class XslCompiledTransformHelper    {        #region Internal Constants        internal const int DefaultBufferSize = 10240; //10 KB        internal const int DefaultThresholdSize = 1048576; //1 MB        internal const string DefaultPartName = "Body";        #endregion        #region Private Static Fields        private static Dictionary<string, MapInfo> mapDictionary;        #endregion        #region Static Constructor        static XslCompiledTransformHelper()        {            mapDictionary = new Dictionary<string, MapInfo>();        }        #endregion        #region Public Static Methods        public static XLANGMessage Transform(XLANGMessage message,                                             string mapFullyQualifiedName,                                             string messageName)        {            return Transform(message,                             0,                             mapFullyQualifiedName,                             messageName,                             DefaultPartName,                             true,                             DefaultBufferSize,                             DefaultThresholdSize);        }        public static XLANGMessage Transform(XLANGMessage message,                                             string mapFullyQualifiedName,                                             string messageName,                                             bool seekOutboundStream)        {            return Transform(message,                             0,                             mapFullyQualifiedName,                             messageName,                             DefaultPartName,                             seekOutboundStream,                             DefaultBufferSize,                             DefaultThresholdSize);        }        public static XLANGMessage Transform(XLANGMessage message,                                             int partIndex,                                             string mapFullyQualifiedName,                                             string messageName,                                             string partName,                                             bool seekOutboundStream,                                             int bufferSize,                                             int thresholdSize)        {            try            {                using (Stream stream = message[partIndex].RetrieveAs(typeof(Stream)) as Stream)                {                    Stream response = Transform(stream, mapFullyQualifiedName, seekOutboundStream, bufferSize, thresholdSize);                    CustomBTXMessage customBTXMessage = null;                    customBTXMessage = new CustomBTXMessage(messageName, Service.RootService.XlangStore.OwningContext);                    customBTXMessage.AddPart(string.Empty, partName);                    customBTXMessage[0].LoadFrom(response);                    return customBTXMessage.GetMessageWrapperForUserCode();                }            }            catch (Exception ex)            {                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);                throw;            }            finally            {                if (message != null)                {                    message.Dispose();                }            }        }        public static XLANGMessage Transform(XLANGMessage[] messageArray,                                             int[] partIndexArray,                                             string mapFullyQualifiedName,                                             string messageName,                                             string partName,                                             bool seekOutboundStream,                                             int bufferSize,                                             int thresholdSize)        {            try            {                if (messageArray != null &&                    messageArray.Length > 0)                {                    Stream[] streamArray = new Stream[messageArray.Length];                    for (int i = 0; i < messageArray.Length; i++)                    {                        streamArray[i] = messageArray[i][partIndexArray[i]].RetrieveAs(typeof(Stream)) as Stream;                    }                    Stream response = Transform(streamArray, mapFullyQualifiedName, seekOutboundStream, bufferSize, thresholdSize);                    CustomBTXMessage customBTXMessage = null;                    customBTXMessage = new CustomBTXMessage(messageName, Service.RootService.XlangStore.OwningContext);                    customBTXMessage.AddPart(string.Empty, partName);                    customBTXMessage[0].LoadFrom(response);                    return customBTXMessage.GetMessageWrapperForUserCode();                }            }            catch (Exception ex)            {                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);                throw;            }            finally            {                if (messageArray != null &&                    messageArray.Length > 0)                {                    for (int i = 0; i < messageArray.Length; i++)                    {                        if (messageArray[i] != null)                        {                            messageArray[i].Dispose();                        }                    }                }            }            return null;        }        public static Stream Transform(Stream stream,                                        string mapFullyQualifiedName)        {            return Transform(stream,                              mapFullyQualifiedName,                              true,                             DefaultBufferSize,                              DefaultThresholdSize);        }        public static Stream Transform(Stream stream,                                       string mapFullyQualifiedName,                                       bool seekOutboundStream)        {            return Transform(stream,                             mapFullyQualifiedName,                             seekOutboundStream,                             DefaultBufferSize,                             DefaultThresholdSize);        }        public static Stream Transform(Stream stream,                                       string mapFullyQualifiedName,                                       bool seekOutboundStream,                                       int bufferSize,                                       int thresholdSize)        {            return Transform(stream,                             new VirtualStream(bufferSize, thresholdSize),                             mapFullyQualifiedName,                             seekOutboundStream,                             bufferSize,                             thresholdSize);        }        public static Stream Transform(Stream inboundStream,                                       Stream outboundStream,                                       string mapFullyQualifiedName,                                       bool seekOutboundStream,                                       int bufferSize,                                       int thresholdSize)        {            try            {                MapInfo mapInfo = GetMapInfo(mapFullyQualifiedName);                if (mapInfo != null)                {                    XmlTextReader xmlTextReader = null;                    try                    {                        xmlTextReader = new XmlTextReader(inboundStream);                        mapInfo.Xsl.Transform(xmlTextReader, mapInfo.Arguments, outboundStream);                        if (seekOutboundStream)                        {                            outboundStream.Seek(0, SeekOrigin.Begin);                        }                        return outboundStream;                    }                    finally                    {                        if (xmlTextReader != null)                        {                            xmlTextReader.Close();                        }                    }                }            }            catch (Exception ex)            {                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);                throw;            }            return null;        }        public static Stream Transform(Stream[] streamArray,                                        string mapFullyQualifiedName)        {            return Transform(streamArray,                             mapFullyQualifiedName,                             true,                             DefaultBufferSize,                             DefaultThresholdSize);        }        public static Stream Transform(Stream[] streamArray,                                       string mapFullyQualifiedName,                                       bool seekOutboundStream)        {            return Transform(streamArray,                             mapFullyQualifiedName,                             seekOutboundStream,                             DefaultBufferSize,                             DefaultThresholdSize);        }        public static Stream Transform(Stream[] streamArray,                                       string mapFullyQualifiedName,                                       bool seekOutboundStream,                                       int bufferSize,                                       int thresholdSize)        {            try            {                MapInfo mapInfo = GetMapInfo(mapFullyQualifiedName);                if (mapInfo != null)                {                    CompositeStream compositeStream = null;                    try                    {                        VirtualStream virtualStream = new VirtualStream(bufferSize, thresholdSize);                        compositeStream = new CompositeStream(streamArray);                        XmlTextReader reader = new XmlTextReader(compositeStream);                        mapInfo.Xsl.Transform(reader, mapInfo.Arguments, virtualStream);                        if (seekOutboundStream)                        {                            virtualStream.Seek(0, SeekOrigin.Begin);                        }                        return virtualStream;                    }                    finally                    {                        if (compositeStream != null)                        {                            compositeStream.Close();                        }                    }                }            }            catch (Exception ex)            {                EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);                throw;            }            return null;        }        #endregion        #region Private Static Methods        private static MapInfo GetMapInfo(string mapFullyQualifiedName)        {            MapInfo mapInfo = null;            lock (mapDictionary)            {                if (!mapDictionary.ContainsKey(mapFullyQualifiedName))                {                    Type type = Type.GetType(mapFullyQualifiedName);                    TransformBase transformBase = Activator.CreateInstance(type) as TransformBase;                    if (transformBase != null)                    {                        XslCompiledTransform map = new XslCompiledTransform();                        using (StringReader stringReader = new StringReader(transformBase.XmlContent))                        {                            XmlTextReader xmlTextReader = null;                            try                            {                                xmlTextReader = new XmlTextReader(stringReader);                                XsltSettings settings = new XsltSettings(true, true);                                map.Load(xmlTextReader, settings, new XmlUrlResolver());                                mapInfo = new MapInfo(map, transformBase.TransformArgs);                                mapDictionary[mapFullyQualifiedName] = mapInfo;                            }                            finally                            {                                if (xmlTextReader != null)                                {                                    xmlTextReader.Close();                                }                            }                        }                    }                }                else                {                    mapInfo = mapDictionary[mapFullyQualifiedName];                }            }            return mapInfo;        }        #endregion    }    public class MapInfo    {        #region Private Fields        private XslCompiledTransform xsl;        private XsltArgumentList arguments;        #endregion        #region Public Constructors        public MapInfo()        {            this.xsl = null;            this.arguments = null;        }        public MapInfo(XslCompiledTransform xsl,                       XsltArgumentList arguments)        {            this.xsl = xsl;            this.arguments = arguments;        }        #endregion        #region Public Properties        public XslCompiledTransform Xsl        {            get            {                return this.xsl;            }            set            {                this.xsl = value;            }        }        public XsltArgumentList Arguments        {            get            {                return this.arguments;            }            set            {                this.arguments = value;            }        }        #endregion    }}

 

In the next episode of the series we will walk through a variety of test scenarios that I used to ensure everything is working as desired, and that also help characterize the performance of the implementation.  Stay tuned for Part 2!

Code

You can immediately download the code of the custom ESB services and test cases here. As always, you are kindly invited to provide feedbacks and comments.

Comments

  • Anonymous
    July 06, 2010
    Hi Paolo, Just a quick comment - at the start of the article you state "Dynamic Send Ports do not support design-time Transport  options provided by Static Send Ports like the Retry Count" - are you sure this is the case? I have found a few posts that seem to suggest that dynamic ports do support retries: social.msdn.microsoft.com/.../48603004-b160-4296-b89c-b5b2a3379338 social.msdn.microsoft.com/.../b5147bab-a5eb-497a-b9a9-9d5401acd2e9 The ESB toolkit seems to support setting properties - are they in fact ignored?

  • Anonymous
    July 06, 2010
    The comment has been removed

  • Anonymous
    November 02, 2012
    Excellent article. Dynamic behavior without dynamic hosts - best of both worlds. I am planning to leverage it in my current application. I see that code link is returning "Forbidden" message. Can you please update or send me source code listed in this site?

  • Anonymous
    November 04, 2012
    Hi Sreeni, thanks for the compliments. Try one of the following links and let me know! http://sdrv.ms/U3tUgo http://sdrv.ms/U3tTce Ciao, Paolo