Introducing the Windows Azure AppFabric Service Bus May 2011 CTP

A lot of partners and customers we talk to are telling us that they think of Service Bus as one of the key differentiators of the Windows Azure platform because it enables customers to build and interconnect applications that reflect the reality of where things stand with regard to moving workloads to cloud infrastructures: Today and for years to come, applications and solutions will straddle desktop and devices, customer-owned and operated servers and datacenters, and private and public cloud assets.

After a decade and more of application integration and process streamlining, no line-of-business application is and should ever again be an island.

If applications move to the cloud or if cloud-based SaaS solutions are to be integrated into enterprise solutions for individual customers, integration invariably requires capabilities like seamless access to services and secure, reliable message flow across network and trust boundaries. Also, as more and more applications are federated across trust boundaries and are built to work for multiple tenants, classic network federation technologies such as VPNs are often no longer adequate since they require a significant degree of mutual trust between parties as they permit arbitrary network traffic flow that needs to be managed.

We just released a new Community Technology Preview that shows that we’re hard at work and committed to expand Service Bus into a universal connectivity, messaging, and integration fabric for cloud-hosted and cloud-connected applications – and we invite you to take a look at our Windows Azure AppFabric SDK V2.0 CTP – May Update and accompanying samples.  

Service Bus is already unique amongst platform-as-a-service offerings in providing a rich services relay capability that allows for global endpoint federation across network and trust boundaries.

As of today, we’re adding a brand-new set of cloud-based, message-oriented-middleware technologies to Service Bus that provide reliable message queuing and durable publish/subscribe messaging both over a simple and broadly interoperable REST-style HTTPS protocol with long-polling support and a throughput-optimized, connection-oriented, duplex TCP protocol.

The new messaging features, built by the same team that owns the MSMQ technology, but on top of a completely new technology foundation, are (of course) integrated with Service Bus’s naming and discovery capabilities and the familiar management protocol surface and allow federated access control via the latest release of the Windows Azure AppFabric Access Control service.  

Queues

Service Bus Queues are based on a new messaging infrastructure backed by a replicated, durable store.

Each queue can hold up to 100MB of message content in this CTP, which is a quota we expect to expand by at least an order of magnitude as the service goes into production. Messages can have user-defined time-to-live periods with no enforced maximum lifetime.

The size of any individual message is limited to 256KB, but the session feature allows creating unlimited-size sequences of related messages whereby sessions are pinned to particular consumers and therefore enabling chunking of payloads of arbitrary sizes. The session state facility furthermore allows transactional recording of the progress a process makes as it consumes messages from a session and we also support session-based correlation, meaning that you can build multiplexed request/reply paths in a straightforward fashion.

Queues support reliable delivery patterns such as Peek/Lock both on the HTTP API and the .NET API that help ensuring processing integrity across trust boundaries where common mechanisms like distributed 2-phase transactions are challenging. Along with that, we have built-in detection of inbound message duplicates, allowing clients to re-send messages without adverse consequences if they’re ever in doubt whether a message has been logged in the queue due to intermittent network issues or an application crash.

In addition to a dead-letter facility for messages that cannot be processed or expire, Queues also allow deferring messages for later processing, for instance when messages are received out of the scheduled processing order and need to be safely put on the side while the process waits for a particular message to permit further progress.

Queues also support scheduled delivery – which means that you can hand a message to the queue infrastructure, but the message will only become available at a predetermined point in time, which is a very elegant way to build simple timers.

Topics

Service Bus Topics provide a set of new publish-and-subscribe capabilities and are based on the same backend infrastructure as Service Bus Queues – and have all the features I just outlined for Queues.

A Topic consists of a sequential message store just like a Queue, but allows for many (up to 2000 for the CTP) concurrent and durable Subscriptions that can independently yield copies of the published messages to consumers.

Each Subscription can define a set of rules with simple expressions that specify which messages from the published sequence are selected into the Subscription; a Subscription can select all messages or only messages whose user-or system defined properties have certain values or lie within certain value ranges. Rules can also include Actions, which allow modifying message properties as messages get selected; this allows, for instance, selecting messages by certain criteria and affinitizing those messages with sessions or to stamp messages with partitioning keys, amongst many other possible patterns.

The filtered message sequence represented by each Subscription functions like a virtual Queue, with all the features of Queues mentioned earlier. Thus, a Subscription may have a single consumer that gets all messages or a set of competing consumers that fetch messages on a first-come-first-served basis.

To name just a few examples, Topics are ideal for decoupled message fan-out to many consumers requiring the same information, can help with distribute work across partitioned pools of workers, and are a great foundation for event-driven architecture implementations.

Topics can always be used just like Queues by setting set up a single, unfiltered subscription and having multiple competing consumers pull messages from the subscription. The great advantage of Topics over Queues is that additional subscriptions can be added at any time to allow for additional taps on the message sequence for any purpose; audit taps that log pre-processing input messages into archives are a great example here.

Access Control Integration

This new CTP federates with the appfabriclabs.com version of the Access Control service, which is compatible with the Access Control “V2” service that is in available commercially since April. The current commercially available version of Service Bus federates with Access Control “V1”.

The Service Bus API to interact with Access Control for acquiring access tokens has not changed, but we are considering changes to better leverage the new federation capabilities of Access Control “V2”.

Customers who are setting up access control rules for Service Bus programmatically will find that there are significant differences between the management APIs of these two versions of the Access Control service. The current plan is to provide a staged migration for customers with custom access control rules on their Service Bus namespaces; migration will be an option for some period of time when we will operate the V1 and V2 versions of the Access Control Service side-by-side. We will publish concrete guidance for this migration over the next several months with initial details coming this week here on this blog.

What Changed and What’s Coming?

We believe that providing these capabilities in the cloud – paired with the features we already have available in Service Bus – will open up a whole new range of possibilities for cloud-hosted and cloud-enhanced applications. We have seen amazing business solutions built on Service Bus and based on customer feedback we’re convinced that the addition of a fully featured set of message-oriented middleware capabilities will enable even more powerful solutions to be built.  Our intention is to make all capabilities contained in this preview commercially available in the second half of 2011.

The load balancing and traffic optimization features for the Relay capability of Service Bus that were added in the PDC’10 CTP of Service Bus have been postponed and are no longer available in this CTP. However, “postponed” does not mean “removed” and we are planning on getting these features back into a CTP release soon. We’ve traded these features for capabilities that we expect will be even more important for many customers: Full backwards compatibility between the current production release of Service Bus and the new version we’re presenting in this CTP, even though we have changed a very significant portion of the Service Bus backend. We are committed to provide full backwards compatibility for Service Bus when the capabilities of this CTP go into production, including backwards compatibility with the Microsoft.ServiceBus.dll that you already have deployed.

To help customers writing apps on platforms other than .NET we will also release Java and PHP samples for the new messaging capabilities in the next few weeks. These samples will be versions of the chat client implemented in the Silverlight and Windows Phone chat samples included in the SDK for this CTP release.

Lastly, and most importantly, the purpose of a Community Technology Preview is to collect feedback from the community.   If you have suggestions, critique, praise, or questions, please let us know at https://social.msdn.microsoft.com/Forums/en-US/appfabricctp/  

You can also Twitter me personally at @clemensv and I’ll let the team know what you have to say.

For the Service Bus Team,

Clemens Vasters
Principal Technical Lead
Windows Azure AppFabric Service Bus