Business Activity Monitoring: GenerateTypedBAMAPI Tool

  • Article

Hi,

For those of you who work with me or have already used this, the “productization” of the GenerateBAMDefinition code has been a long time in the coming!

 

It’s been one of those “back-burner” activities where my proof-of-concept code has held together long enough but as more customers start using it and BizTalk Server 2006 changes required tweaking I’ve finally got round to sorting it out – oh and I needed it for my substantial BAM chapter in my upcoming BizTalk book - blatant plug!

Bring on the GenerateTypedBAMAPI tool! (or for those of you on a certain PoC, bring on the dancing girls :-)

Business Activity Monitoring (BAM) is one of my favourite components of BizTalk as you’ve probably already noticed, it enables rich instrumentation of your BizTalk solution and any systems used either side effectively enabling a enterprise instrumentation solution.

Via an Excel Spreadsheet shipped with BizTalk you define Activities (ultimately become SQL tables) and Activity Items (ultimately become columns within tables), you can then define multiple Views (same concept as SQL views) where you can selectively include data items across multiple activities, rename them and specify measures (pre-calculated items such as Average Age) and dimensions (ways to slice/dice your data), you can even calculate durations between two milestones to perhaps enable timing of remote web service calls for SLA monitoring. Activity Items can either be a Business Milestones (datetime) or Data Items (string, int or decimal).

Once you’ve done this you use the bm.exe tool to create a high performance and scalable database structure from the BAM definition your defined in the Excel Spreadsheet, you get the whole database structure for free! You can specify indexes via the XML file to optimise any querying you may require. Any dimensions and measures you specified are used to create an OLAP cube for you and the DTS jobs are created to populate this cube at regular intervals (all for free!)

The Activities become SQL tables in the database structure and two tables are created per Activity, one called <ActivityName>_Active and one called <ActivityName>_Completed. This partioning means that any queries can be run against the _Completed table and thus not interfere with any other writing going on. The entries are moved from Active to Completed when the Activity is finished with via a SQL trigger.

So we have a database structure, how do we write to it? You can use the out-of-the-box Tracking Profile Editor (TPE) which lets you map Orchestration shapes and items in any BizTalk messages to the “list” of data you defined in the Excel Spreadsheet, this effectively tells BizTalk to subscribe to the events raised when a shape is “invoked” or a message arrives and store this data in the BAM database you created earlier (it does this using the underlying BAM API).

Now, TPE in 2004 was very limited you could only “provide” data to the BAM database you created if it was visible from the Orchestration, i.e. Business Milestones (datetimes) are created by taking System.DateTime.Now when you hit the Orchestration shape in question, and when a BizTalk message arrives any items in that message are stored in the message can be stored as Activity Items, but what if you wanted to store some data that’s only visible inside a Pipeline or a .NET component called from BizTalk? TPE has no visibility of this and so doesn’t offer a way to store this information.

In short TPE is great for a few simplistic scenarios but the good news is there is a full .NET API under the covers (which TPE uses as it happens) which you can use yourself, giving you much finer grained control over what happens and also the ability to store any information regardless of where it’s stored. This is especially true when you have a system that calls BizTalk (or called by) that holds information which would be useful to store alongside any BAM data captured by BizTalk and or course isn’t transmitted to the BizTalk system. For example, a BizTalk solution might call a Credit authorisation Web Service which returns true/false but the Web Service whilst executing has access to other information that might be useful – you could use BAM inside this Web Service (not installed on the BizTalk Server) to store this information

BizTalk Server 2004 didn’t allow you to install the BAM API on non-BizTalk servers without attracting a per-processor license fee for each server, feedback (some driven by me) to the product team resulted in a licensing change which means you can put the BAM API on any server and not pay any license fees (you must have licensed BizTalk somewhere in your organisation however), and have a SQL server for the BAM databases to be created.

This change enables a whole end-to-end instrumentation approach, all driven by BAM…

So the BAM API, most solutions leveraging BAM will end up choosing this over TPE. The API is straight forward but loosely typed (no other way as it has to be general purpose to allow you to create whatever activities and items your like).

You must first “Begin” an Activity which creates the row in the <Activity>_Active table in the BAMPrimaryImport database, you can then “Update” the activity whenever you like before finally “Finishing” your Activity. Begin creates the row ready in the database, Update – updates columns in that row, and Finish “closes” the row off for further updates and causes the trigger to fire which move this row from the Active table to the Completed table.

There are three different types of Event Stream, Direct, Buffered and Orchestration:

The DirectEventStream writes straight to the BAMPrimaryImport database so has the equivalent performance hit (i.e. not good for high volume systems)

The BufferedEventStream which allows multiple writes to be buffered in memory before being stored in binary form to the MessageBox database where it’s then moved into the PrimaryImport database (good for high volume systems where the data doesn’t have to be persisted before continuing)

The OrchestrationEventStream which is used when your calling BAM from an Orchestration, it writes BAM entries to the Persistence Stream of your BizTalk Orchestration so piggy-backs the persistence of your orchestration meaning the performance hit is greatly reduced and it also benefits from the “rollback” behaviour of an Orchestration, so if you write three entries and then roll your orchestration back the entries will be removed unlike the other EventStreams which would require you to manually remove them. The entries are stored in the Message Box as per the BufferedEventStream and then moved off into the BAMPrimaryImport database.

They all share the same interface so are the same to program against except the OrchestrationStream which is a static class so doesn’t have be “newed” each time you wish to use it.

So, consider this BAM API code to write a BAM Activity and some data items:

System.Guid ActivityID = System.Guid.New().ToString();

DirectEventStream es = new DirectEventStream("Integrated Security=SSPI;Data Source=.;Initial Catalog=BAMPrimaryImport",1);

es.BeginActivity(“LoanApplication”, ActivityID);

..

es.UpdateActivity(“LoanApplication”,ActivityID,”Start”,System.DateTime.Now.ToString());

..

es.UpdateActivity(“LoanApplication”,ActivityID,”Name”,”Darren Jefford” );

..

es.UpdateActivity(“LoanApplication”,ActivityID,”End”, System.DateTime.Now.ToString());

..

es.EndActivity(“LoanApplication”,ActivityID);

So fairly straight forward then, but in most projects you end up with many activities containing many items, as you can see the API is loosely typed which means all Activity Names and Activity Items must be specified by using string literals which leads to many potential programming errors, especially if your anything like me and keep spelling Receive wrong. Misnaming any of these items or using the wrong data type causes a Runtime error, and as often happens if you change your BAM Definition as your project progresses you have to keep everything in sync – which to be honest is never going to happen 100% of the time!. Also, a lot of the code is repetitive, BeginActivity, EndActivity, etc.

In the PoC I took part in last year we were faced with 1000s line of code to produce a ful BAM API for the Orchestrations to use, we didn’t have the time for this let alone the changes that happened throughout the week.

So, this is where GenerateBAMDefinitions started, I wrote an XSLT transform that took the BAM XML exported from the spreadsheet to generate a typed BAM API automagically, that way developers would work against classes named the same as activities and properties with named the same as activity items, cutting out any typos in all these string literals, enabling code completition and simplifying the API to Begin, Update and Finish.

I’ve now brought it up to date with a full BizTalk 2006 version, and have written a command line exe that takes a parameter to the Excel Spreadsheet where it extracts the XML for you (no more manual extraction using Excel now), and then generates a typed API for you. Another new addition is the ability to specify the EventStream you wish to use (which wasn’t possible before).

There are no string literals to use and the façade you build against the BAM API is far simpler and potential mistake free, e.g.:

            LoanActivity a = LoanActivity( System.Guid.NewGuid().ToString() );

            a.BeginLoanActivity();

            a.Start = System.DateTime.Now;

    a.Name = "Darren Jefford";

            a.End = System.DateTime.Now;

a.CommitLoanActivity();

            a.EndLoanActivity();

This shaved days of development effort and kept the BAM instrumentation layer working a treat, it also enables you to ensure your build always has the most up-to-date API by allowing you to add your Excel Spreadsheet to your BizTalk solution and setting a pre-build action to run the new GenerateTypedBAMAPI tool whenever the project is built meaning any changes to the BAM definition are instantly reflected in your project.

Everything is wrapped up inside the GenerateTypedBAMAPI tool and I’ve done basic testing of all three Event Stream types and everything seems to work fine, it also copes with the slight API changes between OrchestrationEventStream (static) and the other EventStreams. When you add the generate file to your project ensure that you add a Project Reference to Microsoft.BizTalk.Bam.EventObservation.dll (Buffered and Direct EventStreams) or Microsoft.BizTalk.Bam.XLANGs.dll (OrchestrationEventStream) which you can find in %PROGRAMFILES%\Microsoft BizTalk Server 2006\Tracking.

 

If your run GenerateTypedBAMAPI from the command line it prints out the usage instructions which are pretty straight forward: GenerateTypedBAMAPI.exe <ExcelFile> <OutputCodeFile> Direct|Buffered|Orchestration

If you want to perform Continuation, Relationships or References then you can access the underling EventStream by calling the GetEventStream() method on the BAMDefinition class thus giving you complete flexibility (I’m planning to add native support for these soon, see below).

So, please try it out, especially those that are already using the previous incarnation – if you don’t like the code it generates for whatever reason or wish to lodge “I wish it could generate this..” then please let me know via the Email hyperlink on this blog or via the GotDotNet workspace.

The GotDotNet workspace hosting the tool is located here, Enjoy and please, please let me know any feedback you have (Good or Bad!). The release up there right now is for BizTalk Server 2006, I need to make a few XML namespace and project changes before I can upload the 2004 release but that should be over the weekend, I’ll post a new blog entry when it’s up there.

I’m planning to make the following extensions in the next month or so:

An enumeration of all Activities defined

Continuation Support

Reference support which will expose a typed-method allowing you to choose Activities to link to without having to use the string literal of the Activity Name.

Think of a better way to allow the caller to customise the Connection String (currently hard-coded to the local SQL server – you can of course change this by hand)

Enjoy….

Comments