Throttling Policies and the EWSFindCountLimit

  • Artykuł

One of my favorite Exchange Web Services (EWS) methods is FindItem, primarily because it was the first Web method I wrote when I joined the EWS team long, long ago. Since then, it has undergone lots of optimizations, feature changes, and so on, to make it what it is today. One of the Exchange 2010 changes to FindItem (and FindFolder) is protective in nature – Exchange doesn’t want to drown a Client Access server just because someone decides to pull down the entire contents of their mailbox. This protective change is governed by the EWSFindCountLimit throttling policy parameter.

When a FindItem request comes in and the caller is authenticated, EWS obtains the caller’s budget. Every item (or folder in the case of FindFolder) that EWS processes for the current FindItem request is counted against the budget’s EWSFindCountLimit. As soon as the response is sent back to the caller, the find count charge for the current call is released. Why do we do this? Because items and folders that are waiting to be returned in a response take up memory on the Client Access server.

You will soon discover that budgets have a larger scope than a single request. If a caller makes two concurrent EWS FindItem requests that return 10 items each, the EWSFindCountLimit charge against that caller’s *single* budget is…20. When the first request returns, it will drop to 10 (because those items can now be garbage collected) and then when the second request returns it will drop to 0. It should be relatively obvious that we are trying to limit the caller’s memory footprint on the Client Access server.

What happens, though, when a caller reaches the limit? Well, that depends on whether the caller is being a “good EWS citizen”. And what makes an EWS citizen “good”? Well, in the context of EWSFindCountLimit, a good EWS citizen will do all their FindItem calls by using paging rather than by requesting the entire result set each time. Assume, for a moment, that the EWSFindCountLimit value is 1000 and a bad EWS citizen makes a FindItem call without paging. Let’s further say that the query will return 1001 items. EWS certainly can’t truncate the results, as the caller is expecting, to receive ALL the data back. And EWS can’t turn a non-paged FindItem call into a paged one, as the caller would not expect to check for such a response. So what does EWS do? If the call has a RequestServerVersion that is earlier than Exchange2010, you will receive a failure response with an error code of ErrorServerBusy. If your RequestServerVersion is Exchange2010, you will get back the following response:

<s:Body xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="https://www.w3.org/2001/XMLSchema">
  <m:FindItemResponse xmlns:m="https://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="https://schemas.microsoft.com/exchange/services/2006/types">
    <m:ResponseMessages>
      <m:FindItemResponseMessage ResponseClass="Error">
        <m:MessageText>You have exceeded the maximum number of objects that can be returned for the find operation. Use paging to reduce the result size and try your request again.</m:MessageText>
        <m:ResponseCode>ErrorExceededFindCountLimit</m:ResponseCode>
        <m:DescriptiveLinkKey>0</m:DescriptiveLinkKey>
        <m:MessageXml>
          <t:Value Name="PolicyLimit">1000</t:Value>
        </m:MessageXml>
      </m:FindItemResponseMessage>
    </m:ResponseMessages>
  </m:FindItemResponse>
</s:Body>

Very interesting. For kicks, let’s reduce the EWSFindCountLimit and cycle IIS to pick up the changes (ah yes, the joys of having your own test box to beat upon). Open the Exchange 2010 Management Console as an Exchange Admin and run the following (New-ThrottlingPolicy is only available to administrators, for obvious reasons):

New-ThrottlingPolicy –Name “FooPolicy” –EWSFindCountLimit 1
Set-Mailbox JohnDoe –ThrottlingPolicy FooPolicy
iisreset

Now, on my test box, I created 6 email messages in John Doe’s Drafts folder.  Let’s call FindItem again, but use indexed paging.  Here is my request:

<soap:Header>
    <t:RequestServerVersion Version="Exchange2010"/>
</soap:Header>
<soap:Body>
<FindItem xmlns="https://schemas.microsoft.com/exchange/services/2006/messages"
           xmlns:t="https://schemas.microsoft.com/exchange/services/2006/types" Traversal="Shallow">
    <ItemShape>
        <t:BaseShape>IdOnly</t:BaseShape>
        <t:AdditionalProperties>
            <t:FieldURI FieldURI="item:Subject"/>
        </t:AdditionalProperties>
    </ItemShape>
    <IndexedPageItemView BasePoint="Beginning" Offset="0" MaxEntriesReturned="10000"/>
    <ParentFolderIds>
           <t:DistinguishedFolderId Id="drafts"/>              
        </ParentFolderIds>
</FindItem>

And the response…

<s:Body xmlns:xsi="https://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="https://www.w3.org/2001/XMLSchema">
  <m:FindItemResponse xmlns:m="https://schemas.microsoft.com/exchange/services/2006/messages" xmlns:t="https://schemas.microsoft.com/exchange/services/2006/types">
    <m:ResponseMessages>
      <m:FindItemResponseMessage ResponseClass="Success">
        <m:ResponseCode>NoError</m:ResponseCode>
        <m:RootFolder IndexedPagingOffset="1" TotalItemsInView="6" IncludesLastItemInRange="false">
          <t:Items>
            <t:Message>
              <t:ItemId Id="AAMkA…"/>
              <t:Subject>Message5</t:Subject>
            </t:Message>
          </t:Items>
        </m:RootFolder>
      </m:FindItemResponseMessage>
    </m:ResponseMessages>
  </m:FindItemResponse>
</s:Body>

Whoa – wait a second! There were 6 messages to return, I asked for 10000, but it only returned 1 of them to me! What happened? Well, given that the caller’s EWSFindCountLimit is 1, EWS will NOT return more than 1 record to the user (at a time). However, because the caller was a good Exchange citizen and made a paged request, EWS can “fix up” the result because the user will (read: should) understand that EWS might not return ALL the items. The caller will instead look at the IncludesLastItemInRange attribute and, if it is false, make another FindItem request with the new Offset and continue until IncludesLastItemInRange returns true. If this behavior seems odd to you, consider that this is exactly how the various network streams in the Microsoft .NET framework behave – you continue reading from the stream until the stream reports that it has no more data to read.

So, the moral of the story is: Always use a paging mechanism when calling FindItem or FindFolder. Wisely, the Exchange Web Services Managed API forces you to do so by requiring an ItemView to be passed into FindItem:

service.FindItems(new FolderId(WellKnownFolderName.Drafts), new ItemView(10000));

Now that you know, there is no excuse for failing to use a page view :)

David Sterling
Exchange Web Services
Inside Microsoft Exchange Server 2007 Web Services

Comments

  • Anonymous
    March 23, 2011
    Hi, Can you please give a sample code on how to implement this one? The paging stuff. I'm having problem knowing if there is still available item to read from the response returned by FindItem. This would be a great help. thanks, Ma^

  • Anonymous
    April 07, 2013
    Hi Thom, Thanks for the article.  I have been coding around in circles trying to find the reason for a call to FindItems not returning results as I expect and am still chasing my tail.   I have posted this question social.msdn.microsoft.com/.../cbd1c1b7-c1a0-4a8b-95dc-bcde1eac4870 as I am starting to feel like it may be a bug within Exchange Web Services. Can you cast some light?  Or at least direct me to the best place to get a resolution on this. Thanks Carl

  • Anonymous
    March 04, 2014
    Hi, I have read your article .. and I use it in my project.. the problem is: my EWS project works fine, but every x minutes or hours (in my case I get everyday about half-hour - one hour in 24 hours) I get timeout respond from the exchange server and it happen everyday I send request to server every 5 minutes. Maybe you know why it happen (what a reason of it)?

  • Anonymous
    March 04, 2014
    Forgot to write "Thanks" in previous comment :) So thanks for answers :)

  • Anonymous
    March 04, 2014
    What version of Exchange and is this on-premise or Exchange Online/Office 365?  You might be hitting throttling limits depending on how much you are calling EWS.  When you have put too much pressure on it, it will begin to insert what we call "micro delays" to decrease your throughput.  If on-prem, you should be able to get your Exchange admin to look at the protocol logs to see what (if any) delays were added to your calls and the reason for the long call.

  • Anonymous
    March 05, 2014
    The comment has been removed

  • Anonymous
    March 05, 2014
    Another question:  is it possible that I still can get timeouts if I'll use POP3 or IMAP4 instead using EWS to get emails from server? Thanks again..

  • Anonymous
    March 06, 2014
    Hmmm - I would not try to fetch 10k items in one call.  That could be why your request is timing out.  Instead, try to grab in smaller chunks - maybe 100 at a time and continue fetching until the resulting offset tells you that you are at the end.  Just curious - what is the goal here?  Are you trying to fetch down the entire mailbox?

  • Anonymous
    March 08, 2014
    Thank you very much for your answers :)

  • Anonymous
    March 08, 2014
    Hi David, I'm Yaniv's colleague, thank you for your replies. Yes, we are trying to fetch the entire mailbox, but generally there will be around 10-20 max. items in inbox because every item which is already processed is deleted. So eventuallly although calling for 10k items we are fetching around 10-20  items. When that is the case, you think calling for 100 items only should solve the problem? Thanks again, Orit

  • Anonymous
    April 15, 2014
    Hi Orit and Yaniv, I have the exact same timeout problem and I'm currently calling for 1000 items. I'm pretty sure my problem is related to throtting as I get my timeout in the peak usage hours (backup time and at the beginning of the day when everyone arrives to work) I also process items from a specific folders. So, like you, there can't be many items in the folder since they are moved once they are processed. I use streaming notifications to get new items but at each subscription renewal,, I do a FindItems in the folders to catch messages that may have been missed by the streaming notification (I'm maybe to cautious, but we need to make sure no messages are forgotten. But based on my log, in about a month of production, it hasn't proved useful... maybe I should disable this part.) Did changing to 100 items solved your problems? lynam

  • Anonymous
    March 06, 2015
    The comment has been removed

  • Anonymous
    April 09, 2015
    I have my request limited to 900 by the MaxEntriesReturned but when I call my request with 1001 emails in inbox it still doesn't work

  • Anonymous
    July 08, 2015
    Hello, I've configured an application to use a specific service account to access my users' mailboxes. I've created and associated a throttling policy with this service account (no limit, null values). In reviewing the event logs, I come across an error that seems to suggest the unlimited throttling policy is not being honored. Microsoft.Exchange.Data.Directory.SystemConfiguration.OverBudgetException: This operation exceeds the throttling budget for policy part 'CAS', budget type: 'EWS'. Suggested backoff time 27227 ms. at Microsoft.Exchange.Data.Directory.Budget.CheckOverBudget(CostType costType) at Microsoft.Exchange.Services.Core.PushSubscription.BuildNotification() Does anyone know what (specifically) 'policy part 'CAS', budget type: 'EWS'.' is referring to? Thanks

    • Anonymous
      May 02, 2017
      I have this exact same issue, and cannot locate any documentation about it. The policy has "EWSPercentTimeInCAS" set to "$null", per the vendor recommendation, and we are getting these events (Event ID 22) constantly throughout the day.If you've resolved the issue, or have any further information, I would greatly appreciate it.

  • Anonymous
    June 01, 2017
    The comment has been removed

    • Anonymous
      June 01, 2017
      Correction- I meant ExchangeService object