Data migration and SPO service throttling

We have an excellent support article describing how to avoid getting throttled or blocked in SharePoint Online (link). However, we still receive support requests opened by customers who are moving many items from on-premises SharePoint to SharePoint Online or among different sites within SharePoint Online and notice slowness or throttling. This article pulls together the different scenarios that drive customers to seek support and provides additional perspective.

Scenarios

What triggers customers to contact support regarding slow on-premises-to-cloud or cloud-to-cloud data migration? Although each case is different, these requests tend to fall into a few well-understood scenarios.

Internally-developed CSOM or REST application

SharePoint Online enables client-side access to content and metadata through the Client-Side Object Model (CSOM) and REST APIs. These operations typically are implemented in a PowerShell script or a C# app. If the script or app sends too many requests too quickly, SPO service throttling may cause file uploads to slow or stop completely. In these situations, the script or app must be adjusted so that the account running the script or app doesn’t send too many requests at once.

Third-party app moving data to SPO through the Office 365 Migration APIs

Customers often rely upon third-party applications to manage the migration of data from on-premises SharePoint to SharePoint Online. Best practice is to package data for import into Azure then into SPO through the Office 365 Migration APIs (link). Although the Office 365 Migration APIs are very efficient and not subject to SPO throttling, some third-party migration tools still query CSOM or REST to obtain site and list metadata while moving data from Azure to SPO. Throttling can occur if the migration tool is sufficiently “chatty” when interacting with SPO.

Third-party app moving data within SPO or directly to SPO

Many third-party applications that facilitate on-premises-to-cloud data migration also can move data cloud-to-cloud from one SharePoint Online site to another, either within the same SPO tenant or from one SPO tenant to another. A common example of the latter is moving data from a “pre-production” tenant onto a “production” tenant, with both tenants being owned by the same customer. Also, customers occasionally have reasons for moving data to the cloud directly, bypassing the Office 365 Migration APIs. In these situations, the migration tool must be configured such that each migration account doesn’t send too many requests per second.

Detection

Severe slowness while migrating data tends to be the first indication that throttling is occurring. However, the only certain indicator of SPO service throttling is one or more HTTP 429 responses to requests from the script, app, or migration tool. Please keep in mind that changing or removing the throttling threshold isn’t an option. The throttling threshold is configured at SharePoint Online Farm scope and is enforced to protect service availability. Also, please be aware of published migration speeds when using the Office 365 Migration APIs: https://support.office.com/en-us/article/SharePoint-Online-and-OneDrive-Migration-Speed-21cfb6a0-fa4c-44c6-9a39-0f45e85e371f.

Solutions

Here is a summary of the guidance we’ve provided to customers who have reported these issues. Much of this information expands upon the following support article: How to avoid getting throttled or blocked in SharePoint Online.

Guidance for customer-developed apps and scripts

Any PowerShell script, C# app, or 3^rd party tool that bulk-copies or bulk-moves data to and from SPO must guard against HTTP 429 responses. Here are guidelines and documentation that should help.

• Most crucially, ensure a steady-state request rate of 1-2 requests per second. https://dev.office.com/sharepoint/docs/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online
• Implement the “incremental back-off and retry” pattern, which pauses for some duration after observing an HTTP 429 response. https://msdn.microsoft.com/en-us/pnp_articles/handle-sharepoint-online-throttling-by-using-exponential-back-off
• If writing a PowerShell script, consider using the OfficeDev PnP PowerShell module, which internally catches and handles HTTP 429 throttling errors. https://github.com/SharePoint/PnP-PowerShell

Guidance for customers running third-party data migration apps

If the data migration app you are using doesn’t have configuration options that match these recommendations, contact the migration software vendor for support.

• Use more than one Office 365 account to perform the automated data migration. Customers typically use between 2 and 8 accounts, so perhaps 4 accounts would be a good starting point depending upon license availability.
• Maintain a steady-state request rate of 1-2 requests per second per account. Keeping the request rate close to 1 per second per account minimizes the risk of throttling.
• As previously mentioned, we often see that much of the traffic generated by migration software isn't actual file transfers but rather “chatty” requests for metadata (/vti_bin/sites.asmx, /vti_bin/lists.asmx) or CSOM calls (/vti_bin/client.svc/processquery). In some cases, these additional requests have breached the requests per second per account guideline. When this occurs, we recommended that the customer contact the migration software vendor for support.

What to expect if you contact Microsoft for support

As mentioned, throttling isn’t reduced or disabled for any customer under any circumstances. If you contact Microsoft for support, we can confirm whether your tenant has been throttled over the past week. Once we confirm throttling, we will provide guidance like the above and provide links to the articles above. Also, we may ask you to record a Fiddler trace of the data migration, which will confirm “chattiness” and clearly show any HTTP 429 responses.