BizTalk: Configuring Throttling to slow down message processing

  • Article

Problem

Normally you want messages in BizTalk to be processed as fast as possible. However, this is not always the case, especially when BizTalk delivers messages faster than the receiving system can handle. I have seen this behaviour with systems like IBM ContentManager, which crashes hard when BizTalk pounds it with thousands of messages in short periods. Sometimes you actually like BizTalk to slow down, and process messages at a more moderate pace. So what do you do? Develop a custom library in C# to control the load? No!

Solution

BizTalk hosts have a wide variety of configuration options, and can be tuned for specific scenarios like low latency. The built-in throttling functionality will kick in when BizTalk cannot handle the load, but this behavior can also be forced. By changing the Resource-Based Throttling and Rate-Based Throttling settings, we can force a host to process messages slowly.

The rest of this article will discuss this configuration file in the TechNet Gallery (SlowlyRunningHost.xml).

Details

The XML configuration file contains several settings, which change the default behaviour of a BizTalk host. The host name in this case is SlowlyRunningHost, and the following details and tests are performed on a standard BizTalk VM running in Azure. An experienced BizTalk administrator should have no problem reproducing this test based on the descriptions and pictures. If you need assistance, view the How to reproduce this test section below, which explains everything step by step.

Figure 1. SlowlyRunningHost in BizTalk Administration Console

To make the host run slowly, we need to alter Resource-Based Throttling settings. To read and modify these settings, right-click your BizTalk Group in the Administration Console, choose Settings... and then Hosts. Select the SlowlyRunningHost in the dropdown. The "i" icon to the right indicates the value has been changed. For a thorough explanation of all settings, read this article.

Resource-Based Throttling - Configuration Changes

  • Threads: Maximum number of messaging engine threads per CPU. Default value is 0, meaning no threshold
  • In-process messages: Maximum number of in-memory in-flight messages allowed before throttling message delivery begins. Default value is 1000
  • Internal message queue size: Size of the in-memory queue. This queue serves as a temporary placeholder for delivering messages. Default value is 100
  • Message count in DB: Maximum number of items in the database. Default value is 50000
  • Severity - Inflight message: Controls the reaction time of throttling when the in-process size exceeds the threshold. Default value is 75%

Figure 2. Resource-Based Throttling

Next, we need to change Rate-Based Throttling settings for the host.

Rate-Based Throttling - Configuration Changes

  • Publishing
    • Minimum number of samples: The number of samples used to determine the rate of message publishing by the service classes of the host. Default value is 100
    • Sampling window duration: Time-window beyond which samples will not be considered. Default value is 15s
    • Rate overdrive factor: Percentage factor by which the system will overdrive the input rate. Default value is 125%
    • Maximum throttling delay: The maximum delay imposed for message publishing throttling. Default value is 300s
    • Throttling override: Can be used to override message publishing throttling. Default value is Do Not Override
  • Delivery
    • Minimum number of samples: The number of samples used to determine the rate of message delivery by the service classes of the host. Default value is 100
    • Sampling window duration: Time-window beyond which samples will not be considered. Default value is 15s
    • Rate overdrive factor: Percentage factor by which the system will overdrive the input rate. Default value is 125%
    • Maximum throttling delay: The maximum delay imposed for message delivery throttling. Default value is 300s
    • Throttling override: Can be used to override message delivery throttling. Default value is Do Not Override

Figure 3. Rate-Based Throttling

Configuration Settings XML-file

This is the exported XML configuration file which corresponds to the settings explained below. Note that settings not related to the SlowlyRunningHost host have been removed.

<?xml version="1.0" encoding="utf-8"?>
<Settings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <ExportedGroup>
  </ExportedGroup>
  <GroupSettings>
  </GroupSettings>
  <HostSettings>
    <Host Name="SlowlyRunningHost">
      <Setting Name="AllowMultipleResponses">False</Setting>
      <Setting Name="AuthTrusted">False</Setting>
      <Setting Name="DBQueueSizeThreshold">1000</Setting>
      <Setting Name="DBSessionThreshold">0</Setting>
      <Setting Name="DehydrationBehavior">2</Setting>
      <Setting Name="DeliveryQueueSize">1000</Setting>
      <Setting Name="GlobalMemoryThreshold">0</Setting>
      <Setting Name="HostTracking">False</Setting>
      <Setting Name="InflightMessageThreshold">3</Setting>
      <Setting Name="IsHost32BitOnly">False</Setting>
      <Setting Name="LegacyWhitespace">False</Setting>
      <Setting Name="MessageDeliveryMaximumDelay">10000</Setting>
      <Setting Name="MessageDeliveryOverdriveFactor">125</Setting>
      <Setting Name="MessageDeliverySampleSpaceSize">1</Setting>
      <Setting Name="MessageDeliverySampleSpaceWindow">3000</Setting>
      <Setting Name="MessagePublishMaximumDelay">10000</Setting>
      <Setting Name="MessagePublishOverdriveFactor">100</Setting>
      <Setting Name="MessagePublishSampleSpaceSize">1</Setting>
      <Setting Name="MessagePublishSampleSpaceWindow">30000</Setting>
      <Setting Name="MessagingMaxReceiveInterval">500</Setting>
      <Setting Name="MessagingReqRespTTL">20</Setting>
      <Setting Name="MsgAgentPerfCounterServiceClassID">0</Setting>
      <Setting Name="ProcessMemoryThreshold">25</Setting>
      <Setting Name="SubscriptionPauseAt">0</Setting>
      <Setting Name="SubscriptionResumeAt">0</Setting>
      <Setting Name="ThreadPoolSize">20</Setting>
      <Setting Name="ThreadThreshold">3</Setting>
      <Setting Name="ThrottlingBatchMemoryThresholdPercent">1</Setting>
      <Setting Name="ThrottlingDeliveryOverride">1</Setting>
      <Setting Name="ThrottlingDeliveryOverrideSeverity">100</Setting>
      <Setting Name="ThrottlingLimitToTriggerGC">80</Setting>
      <Setting Name="ThrottlingPublishOverride">1</Setting>
      <Setting Name="ThrottlingPublishOverrideSeverity">100</Setting>
      <Setting Name="ThrottlingSeverityDatabaseSize">1</Setting>
      <Setting Name="ThrottlingSeverityInflightMessage">10</Setting>
      <Setting Name="ThrottlingSeverityProcessMemory">500</Setting>
      <Setting Name="ThrottlingSpoolMultiplier">10</Setting>
      <Setting Name="ThrottlingTrackingDataMultiplier">10</Setting>
      <Setting Name="TimeBasedMaxThreshold">1800</Setting>
      <Setting Name="TimeBasedMinThreshold">1</Setting>
      <Setting Name="UseDefaultAppDomainForIsolatedAdapter">False</Setting>
      <Setting Name="XlangMaxReceiveInterval">500</Setting>
    </Host>
  </HostSettings>
  <HostInstanceSettings>
  </HostInstanceSettings>
</Settings>

Important notes

  • This test uses one host for receiving and sending. In a production environment, you should implement a host separation strategy. Read more about this here
  • The values here may need tuning based on your needs and requirements. They may also cause different test results, which vary with different hardware and software configurations. Thorough testing is vital
  • Test results vary between the first and subsequent batch of messages after host instance restart

How to reproduce this test

  1. Create a BizTalk VM in Azure, or use a different test environment [How?]
    • BizTalk 2013 on Windows Server 2012
    • Use Standard A2 tier
  2. Create a host called SlowlyRunningHost [How?]
  3. Create a host instance using the host from step 2 [How?]
  4. Download the configuration file from the TechNet Gallery, and copy to your BizTalk server
  5. Backup the existing configuration [How?]
  6. Import the configuration file from step 4 [How?]
  7. Create Send and Receive handlers for the new host for the FILE adapter [How?]
  8. Create a new messaging application which moves all files from one folder to another. Use SlowlyRunningHost
  9. Optional: Reduce Batch Size to 1 [How?]
  10. Restart the host instance [How?]

Analysis

This section will illustrate how the host processes messages after implementing the tuning explained above. The application is moving 30 XML-files from one folder to another on the local C-drive of the server. The pipelines are PassThruReceive and PassThruTransmit. The file sizes vary between 11KB and 2336KB.

  • When using the standard BizTalkServerApplication host (with default settings), this process completes in a couple of seconds
  • When using the SlowlyRunningHost, this process completes in about 5 minutes

BizTalk Administration Console

During the test we can see that BizTalk is processing a small amount of messages at the same time, in this case 2. This is as expected.

Figure 4. All In-Progress Service Instances

Performance Monitor

Running Performance Monitor during the test gives us a lot of information. See screenshot below. Some key observations:

  • In-process message count is 2, and In-process message count threshold is 3, which corresponds to the In-process messages count setting in BizTalk
  • Message delivery delay (ms) and Message publishing delay (ms) is 10000, which corresponds to the Maximum throttling delay setting in BizTalk
  • Message delivery throttling state is 10, and Message publishing throttling state is 11, which means custom/user override throttling
  • Thread count is 56, and Thread count threshold is 3, which incurs throttling and corresponds to the Threads setting in BizTalk

Figure 5. Performance Monitor during test

Performance Analysis of Logs (PAL)

Running PAL during tests provides a lot of details and graphical overview of important performance counters. A few of them will be used here as examples.

Message publishing throttling state: This is 11 during the whole test (over override on publishing)

Figure 6. Message publishing throttling state

Message delivery throttling state: This is 10 during the whole test (over override on delivery)

Figure 7. Message delivery throttling state

Message delivery delay (ms): This is 10000 (10 seconds) during the whole test.

Figure 8. Message delivery delay (ms)

Other Languages

This article is also available in the following languages

See Also

Suggested reading following resources:

Another important place to find an extensive amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.