BizTalk Server: Dynamic Schema Resolver Real Scenario

Article
1/17/2024

Introduction

As a BizTalk consultant, you had an assignment that a service provider asked you to enhance the existing BizTalk solution.

You checked the current implementation and you found that the existing BizTalk solution as shown in Figure 1

**Figure 1 **- Current BizTalk Solution Architecture

They are receiving different flat files from different customers and each customer has a different flat file format. All customers send these flat files in one receive path with a specific unique file name format ####YYYYMMDDHHMMSSNNN.txt

The first 4 digits are alphanumeric of customer code.

Each customer publishes a message to BizTalk Engine and other send ports subscribe these messages.

In this article, I will focus on the dynamic resolver engine which is specifically in receive and send pipelines

Bad Solution

The implementation required the following steps to add a new customer to the system:

Adding New Source Steps

Add a new flat file schema matching the structure of a new customer as shown in Figure 2

**Figure 2 **- Sample Source Schema

Figure 1- Current BizTalk Solution Architecture
Add a new receive pipeline, add flat file disassembler and set Document Schema property to Flat file schema the created in step 1 as shown in Figure 3

**Figure 3 **- Receive Pipeline
Add a new receive location and set Receive Pipeline to the new receive pipeline that created in step 2 as shown in Figure 4

**Figure 4 **- Set Receive Pipeline to receive location
Click on configure button and set File mask as shown Figure 5

**Figure 5 **- Set File Mask

In case there is a new Destination then you need to do the following steps

Adding New Destination Steps

Add a new flat file schema matching the structure of a new destination as shown in Figure 6

**Figure 6 **- Sample Destination Schema
Add a new send pipeline , add flat file assembler and set Document Schema property to Flat file schema the created in step 1 as shown in Figure 7

Figure 7 - Send Pipeline
Add a new map between source and destination like sample as shown in Figure 8

Figure 8 - Sample Map between source and destination
Add a new Send Port in BizTalk Administration as show in Figure 9

**Figure 9 **- Send Pipeline in Send port
Click configure and set file name format as shown in Figure 10

**Figure 10 **- Setting File Name Format
Select Outbound Maps and select the correct map as shown in Figure 11

**Figure 11 **- Setting Outbound Map
Select Filters and Set BTS.ReceivePortName == Rcv_Messages which is the name of receive port as show in Figure 12

**Figure 12 **- Setting Filter in send port

Problem

Now imagine that you want to add 100 customers as a source then you need to add 100 of Receive Pipelines artifacts and you need to repeat steps of 2-4 of New Source Steps section 100 times. Another issue, imagine that you want to add a new 100 destination then you need to create 100 of Send Pipelines artifacts and you need to repeat steps 2-7 of New Destination Steps section 100 times.

Good Solution

We need to implement a dynamic schema resolver which is a mechanism of dynamically associate message types to inbound instances based on an identifier contained in each instance message.

We need to build a custom pipeline that takes a message type from configuration database, like Business Rule Engine or using your custom configuration database table and set Document Schema to dis-assembler using the unique key that distinguishes your source message, you can use context properties of the message or using any unique data of the content of the message.

In our case the unique key is the customer code which is the first 4 digits of the file name

For send custom pipeline, we have the xml message inside BizTalk engine which contains the schema type, we just need to set document schema into assembler at runtime

Figure 13 shown the Good solution architecture

**Figure 13 **- Good Solution Architecture using Dynamic Resolver Mechanism

We need to do the following implementation steps

Creating BRE Policy

Create new schema with 2 element one for Source Code and one for message type(Namespace#RootNode) as shown in **Figure 14 **

**Figure 14 **- BRE Schema
Create a new Policy SchemaResolverPolicy and add a rule for each source as shown in Figure 15

**Figure 15 **- Sample BRE Policy

Create Custom Pipeline Component

We need to create a custom pipeline for receiving and sending pipeline as the following steps

Create Custom Receive Pipeline

Add a new class library project as shown in Figure 16

**Figure 16 **- Add new class library to contain pipeline components
Add a signature to the assembly, to allow assembly to be put into the GAC. To do this, right-click the new project and choose Properties. Then go to the Signing tab, as shown in Figure 17

**Figure 17 **- Adding a signature to the class library
To create classes that BizTalk recognizes as pipeline components, you need to implement some specific interfaces. These interfaces are specified in the Microsoft.BizTalk.Pipeline namespace. You therefore need to reference the assembly that contains this namespace and these interfaces. Right-click your project or on the References node in your project and click Add Reference. Go to the Browse pane and browse your way to <InstallationFolder>\Microsoft.BizTalk.Pipeline.dll and add this as a reference
To delegate call to Probe method which is implemented to investigate the message to decide if message is recognizable or not. If the message is recognizable then it returns “True” otherwise “False”. We need to add <InstallationFolder>\Microsoft.BizTalk.Pipeline.Components.dll.
To call BRE api we need to reference to <installation drive>:\Program Files\Common Files\Microsoft BizTalk\Microsoft.RuleEngine.dll
Delete the Class1.cs file that was automatically created for you and instead add a new class to your project by right-clicking the project and choosing Add, New Item. Choose Class as the type and provide a name for your class as show in Figure 18

**Figure 18 - ** Adding a new class to your class library.
Mark your class as a public and decorate it with some attributes Guid telling the pipeline designer how the component can be used in our receiving custom pipeline we need to use:

CategoryTypes.CATID_PipelineComponent This category does not affect which stages a component can be placed in, but merely serves as an indication for the pipeline designer that this is a pipeline component.

CategoryTypes.CATID_DisassemblingParser Components in this category can be placed in the Disassemble stage of a receive pipeline and we are assigning document schema in disassemble stage. To use the attribute you need to use namespace Microsoft.BizTalk.Component.Interop as show in Figure 19

using Microsoft.BizTalk.Component.Interop;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
 
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    public class  SchemaResolverFlatFileDasmComp
    {
    }
}

**Figure 19 **- Adding Pipeline attributes

Decorate your class with GUID attribute by copy a new guid from TOOLS Create GUID As shown in Figure 20 and add namespace System.Runtime.InteropServices.

**Figure 20 **- Create new GUID

All custom pipeline components must implement the following two interfaces:

IComponentUI, which lets you specify an icon for the component and a method for validating any properties the user can set values in
IBaseComponent, which basically lets you specify three properties that the pipeline designer uses to get a name, version, and description from your component to show in the designer As shown in Figure 21

using Microsoft.BizTalk.Component.Interop;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
    public class  SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI
    {
        #region IBaseComponent Members
        public string  Description
        {
            get
            {
                return "Flat file disassembler which resolves schemas using message context";
            }
        }
        public string  Name
        {
            get
            {
                return "Receive Pipeline Schema Resolver";
            }
        }
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }
        #endregion
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
    }
}

**Figure 21 **- Implement IBaseComponent, IComponentUI interfaces

Implement interface IDisassemblerComponent which exposes two methods needed for a disassembler as shown in Figure 22

using Microsoft.BizTalk.Component;
using Microsoft.BizTalk.Component.Interop;
using Microsoft.BizTalk.Message.Interop;
using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
    public class  SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent
    {
        #region IBaseComponent Members
        public string  Description
        {
            get
            {
                return "Flat file disassembler which resolves schemas using message context";
            }
        }
        public string  Name
        {
            get
            {
                return "Receive Pipeline Schema Resolver";
            }
        }
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }
        #endregion
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
 
        private FFDasmComp  disassembler = new FFDasmComp();
        #region IDisassemblerComponent Members
        public IBaseMessage GetNext(IPipelineContext pContext)
        {
            // Delegate call to Flat File disassembler
            return disassembler.GetNext(pContext);
        }
 
        public void  Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Delegate call to Flat File disassembler
            disassembler.Disassemble(pContext, pInMsg);
        }
        #endregion      
    }
}

**Figure 22 **- Implement IDisassemblerComponent interface

Implement interface IProbeMessage which will implement Probe method which called, and the component is then executed only if the Probe method returns a value of True and this will do the functionality of schema resolver. We read ReceiveFileName context property from message context to read the first 4 digits of the file name to retrieve schema type from BRE as show in Figure 23

using Microsoft.BizTalk.Component;
using Microsoft.BizTalk.Component.Interop;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.RuleEngine;
using System;
using System.Collections;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Runtime.InteropServices;
using System.Text;
using System.Threading.Tasks;
using System.Xml;
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
    [ComponentCategory(CategoryTypes.CATID_DisassemblingParser)]
    [Guid("B661FCC5-1EBB-4FA8-A0F9-B7AA66EDE0E7")]
    public class  SchemaResolverFlatFileDasmComp : IBaseComponent, IComponentUI, IDisassemblerComponent ,IProbeMessage 
    {
        private FFDasmComp  disassembler = new FFDasmComp();
        private static  Dictionary<string, string> cachedSources = new  Dictionary<string, string>();
 
        #region IBaseComponent Members
        public string  Description
        {
            get
            {
                return "Flat file disassembler which resolves schemas using message context";
            }
        }
        public string  Name
        {
            get
            {
                return "Receive Pipeline Schema Resolver";
            }
        }
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }
        #endregion
 
        #region IComponentUI Members
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
 
        #region IDisassemblerComponent Members
        public IBaseMessage GetNext(IPipelineContext pContext)
        {
            // Delegate call to Flat File disassembler
            return disassembler.GetNext(pContext);
        }
 
        public void  Disassemble(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Delegate call to Flat File disassembler
            disassembler.Disassemble(pContext, pInMsg);
        }
        #endregion  
     
        #region IProbeMessage Members
        public bool Probe(IPipelineContext pContext, IBaseMessage pInMsg)
        {
            // Check arguments
            if (null ==  pContext)
                throw new  ArgumentNullException("pContext");
 
            if (null ==  pInMsg)
                throw new  ArgumentNullException("pInMsg");
 
            // Check whether input message doesn't have a body part or it is set to null, fail probe in those cases
            if (null ==  pInMsg.BodyPart || null  == pInMsg.BodyPart.GetOriginalDataStream())
                return false;
 
            Stream sourceStream = pInMsg.BodyPart.GetOriginalDataStream();
            string messageType = string.Empty;
            //Read ReceivedFileName from the message context
            string srcFileName = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties").ToString();
            srcFileName = Path.GetFileName(srcFileName);
            //Substring the first four digits to take source code to use to call BRE API
            string customerCode = srcFileName.Substring(0, 4);
            //create an instance of the XML object
            XmlDocument xmlDoc = new  XmlDocument();
            xmlDoc.LoadXml(string.Format(@"<ns0:Root xmlns:ns0='http://TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE'>
                          <SrcCode>{0}</SrcCode>
                          <MessageType></MessageType>
                        </ns0:Root>", customerCode));
            //retrieve source code in case in our cache dictionary 
            if (cachedSources.ContainsKey(customerCode))
            {
                messageType = cachedSources[customerCode];
            }
            else
            {
                TypedXmlDocument typedXmlDocument = new  TypedXmlDocument("TechNetWiki.SchemaResolver.Schemas.SchemaResolverBRE", xmlDoc);
                Microsoft.RuleEngine.Policy policy = new  Microsoft.RuleEngine.Policy("SchemaResolverPolicy");
                policy.Execute(typedXmlDocument);
                XmlNode messageTypeNode = typedXmlDocument.Document.SelectSingleNode("//MessageType");
                messageType = messageTypeNode.InnerText;
                policy.Dispose();
                // Fail if message type is unknown
                if (string.IsNullOrEmpty(messageType))
                    return false;
                cachedSources.Add(customerCode, messageType);
            }
 
            // Get document spec from the message type
            IDocumentSpec documentSpec = pContext.GetDocumentSpecByType(messageType);
            // Write document spec type name to the message context so Flat File disassembler could access this property and
            // do message processing for a schema which has document spec type name we've discovered
            pInMsg.Context.Write("DocumentSpecName", "http://schemas.microsoft.com/BizTalk/2003/xmlnorm-properties", documentSpec.DocSpecStrongName);
            // Delegate call to Flat File disassembler
            return disassembler.Probe(pContext, pInMsg);
 
        }
        #endregion
    }
}

**Figure 23 **- Implement IProbeMessage interface

Create Custom Send Pipeline

Add a new Item class to your class library project and rename it
Do same steps of creating custom receive pipeline class but for sending custom pipeline we need to use CategoryTypes.CATID_AssemblingSerializer attribute and implement IAssemblerComponent interface as show in Figure 24

using Microsoft.BizTalk.Component.Interop;
using System;
using System.Runtime.InteropServices;
using Microsoft.BizTalk.Message.Interop;
using Microsoft.BizTalk.Component;
using System.Collections;
 
namespace TechNetWiki.SchemaResolver.PipelineComponent
{
    [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]    
    [ComponentCategory(CategoryTypes.CATID_AssemblingSerializer)]
    [Guid("ACF451CE-E60D-42E4-8AB0-ED4CADD8B23E")]
    public class  SchemaResolverFlatFileAsmComp : IBaseComponent, IComponentUI, IAssemblerComponent
    {
        System.Collections.Queue qOutputMsgs = new  System.Collections.Queue();
        private FFAsmComp assembler = new FFAsmComp();
 
        #region IBaseComponent members       
        public string  Description
        {
            get
            {
                return "Flat file assembler which resolves schemas using message context";
            }
        }
              
        public string  Name
        {
            get
            {
                return "Send Pipeline Schema Resolver";
            }
        }
            
        public string  Version
        {
            get
            {
                return "1.0";
            }
        }       
        #endregion
 
        #region IComponentUI Members
 
        public IntPtr Icon
        {
            get
            {
 
                return System.IntPtr.Zero;
            }
 
        }
        
        public IEnumerator Validate(object obj)
        {
            return null;
        }
        #endregion
 
        #region IAssemblerComponent Members
        public void  AddDocument(IPipelineContext pContext, Microsoft.BizTalk.Message.Interop.IBaseMessage pInMsg)
        {
            qOutputMsgs.Enqueue(pInMsg);
        }
 
        public IBaseMessage Assemble(IPipelineContext pContext)
        {
            IBaseMessage msg = (IBaseMessage)qOutputMsgs.Dequeue();           
            assembler.AddDocument(pContext, msg);            
            return assembler.Assemble(pContext);
        }
        #endregion
    }
}

**Figure 24 **- Custom Send Pipeline component code

Create Receive and Send Pipeline artifacts

Build you Class library project and copy the dll file and paste it to this location <InstallationFolder>\Pipeline Components
Add new receive pipeline to your BizTalk project
Right Click on Toolbox then select choose items then choose as show in Figure 25

**Figure 25 **- Selecting Custom Pipeline components
Drag Receive Pipeline Schema Resolver component from Toolbox and drop it to Disassemble part as shown in Figure 26

**Figure 26 **- Drag Drop Receive Pipeline Schema Resolver component
Add new send pipeline to your BizTalk project
Drag Send Pipeline Schema Resolver component from Toolbox and drop it to assemble part as shown in Figure 27

**Figure 27 **- Drag Drop Send Pipeline Schema Resolver component
Deploy You Pipeline BizTalk project then open BizTalk Administration and create new Receive location in your BizTalk application, configure the location and select the receive pipeline of schema resolver as shown in Figure 28

**Figure 28 **- Select receive schema resolver pipeline
Create new send port, select file adapter,configure the location and select send pipeline as shown in Figure 29

**Figure 29 - **Select send schema resolver pipeline
Select all required Outbound maps to map files from source to destination as shown in Figure 30

**Figure 30 **- Select Outbound maps
Filter your send port to subscribe from receive port as show in Figure 31

**Figure 31 **- Filter send port

We can realized now that if we want to add any new source or any new destination, you just need to add schemas, maps and configure BRE or your configuration table.
This means that you reduced the number of artifacts in your solution pipelines and receive and send ports.
In the bad solution if you have 100 sources and 100 destinations, then you will have 100 receive locations, 100 send ports and 100 pipelines.
However, in the good solution you will have 1 receive location, 1 send port and 2 pipelines and your configuration table.

One more thing, that we can do more enhancement to our solution by passing parameter generically to BRE instead of depending on the first 4 characters of file name,
we can use any property of the message context, but for demonstration purposes I just use the file name.

Moreover, we can use canonical schema and mapping (inbound and outbound mapping) dynamically which I will write another article about this subject later on.

Sample Source Code

You can find the sample source code in the following link sample source code

Conclusion

This article demonstrated

The real scenario of receiving and send Flat File from different sources with different formats and sending files to different destinations with different formats
How to implement this scenario using the static BizTalk solution
How to implement the schema resolver mechanism using custom receiving and sending pipeline, which made our solution more dynamic and easy to maintain
How to build custom receive and send pipeline in details
Benefits of using dynamic schema resolver mechanism
Call BRE API from C# code

Share via