OneNote API calls fail with a large number of items in a SharePoint document library

Article
09/11/2016

Hello world! Paresh Moradiya and Sri Srinivasan, members of the OneNote API team here. We've heard from some of you saying that you're running into issues when accessing SharePoint-hosted notebooks in very large document libraries. We have come up with some instructions on how you can eliminate these OneNote API failures.

The problem

Applications call the OneNote REST API to create or access OneNote content in Office 365. Sometimes the OneNote API calls can fail when they are executed against SharePoint document libraries that contain a large number of items (items can be folders or files, some of which could be OneNote entities such as notebooks, sections or section groups).

When working with SharePoint document libraries that contain a large number of items, the OneNote API service is subject to throttling measures implemented by SharePoint for the purposes of regulating resource usage. Under these circumstances, the user of the application may see an application-specific error. The application calling the OneNote API will itself receive one of the errors below:

{
"error": {
"code": "10008",
"message": "One or more of the document libraries on the user or group's OneDrive contains more than 5,000 OneNote items (notebooks, sections, section groups) and cannot be queried using the API. Please make sure that none of the user or group's document libraries contains more than 5,000 OneNote items. Please follow the link below for instructions on how to remedy this situation.",
"@api.url": "https://aka.ms/onenote-errors#C10008"
}

{
"error": {
"code": "10013",
"message": "One or more of the document libraries on the user or group's OneDrive contains more than 20,000 items and cannot be indexed for querying using the API. Please follow the link below for instructions on how to remedy this situation.",
"@api.url": "https://aka.ms/onenote-errors#C10013"
}

The solution

To mitigate the OneNote API failures described above, you must first assess the current state of the SharePoint document libraries on the SharePoint site and then take a few remedial steps based on the current state.

Assess the current state

Download the OneNote API Diagnostics tool available here and follow the steps below.

Enter the following parameters into the tool

Parameter	Description
SharePoint URL	The SharePoint site's URL. For example, a user's personal site might use the format `https://<<Your Tenant>>-my.SharePoint.com/personal/<<Your email address - Replace all "." with "_">>` (contact your administrator for the format of your personal site's URL) You can also enter the URL for a non-personal SharePoint site
User	Your Office 365 user name (email address)
Password	Your office 365 password

Click the "Get Info" button. This will fetch the diagnostic information from the specified SharePoint site. The operation may take a few seconds to process.

NOTE: The tool processes only the default document library on the SharePoint site by default (for a user's personal site, this is the user's OneDrive for Business). If the API issues above are not resolved by just targeting the default document library, you can choose the tool's option to target all document libraries on the SharePoint site.

Take appropriate remedial steps

Based on the output of the diagnostics tool, you can take the remedial steps suggested below. These steps are applicable at a document library level. If you chose to target all document libraries on the SharePoint site, you will need to execute the remedial steps below on all the document libraries.

Case 1: Items Count is greater than 20,000

If Items Count is greater than 20,000 and File Type = false or HTML File Type = false, then you need to bring the item count in the document library to under 20,000. You can do this by moving items out of the document library to a different storage location (such as another user or group's OneDrive). You can also delete the items that are not required anymore.

After you have reduced the item count to under 20,000, rerun the tool to make sure the items count is less than 20,000. Click the "Build Index" button so the required indexes are built for the OneNote API to work properly.

Case 2: Sections Count is greater than 5,000

If Sections Count is greater than 5,000, then you need to bring the section count in the document library to under 5,000. You can do this by moving some of your notebooks out of the document library to a different storage location (such as another user or group's OneDrive). You can also delete the notebooks that are not required anymore.

After you have reduced the section count to under 5,000, rerun the tool to make sure the sections count is less than 5,000. Ideally you should keep this number below 5,000 so that you can have enough room to create new sections. If the sections count in the document library crosses 5,000 again, you will start seeing the OneNote API failures and you will need to come back here and take the same remedial steps.

Case 3: Folders Count is greater 5,000

If Folders Count is greater than 5,000, then you need to bring the folder count in the document library to under 5,000. You can do this by reorganizing the content in the document library and eliminating folders. You can also delete the folders that are not required anymore.

After you have reduced the folder count to under 5,000, rerun the tool to make sure the folder count is less than 5,000. Ideally you should keep this number below 5,000 so that you can have enough room to create new folders. If the folder count in the document library crosses 5,000 again, you will start seeing the OneNote API failures and you will need to come back here and take the same remedial steps.

Case 4: Notebooks Count is greater than 5,000

If Notebooks Count is greater than 5,000, then you need to bring the notebook count in the document library to under 5,000. You can do this by moving some of your notebooks out of the document library to a different storage location (such as another user or group's OneDrive). You can also delete the notebooks that are not required anymore.

After you have reduced the notebook count to under 5,000, rerun the tool to make sure the notebook count is less than 5,000. Ideally you should keep this number below 5,000 so that you can have enough room to create new notebooks. If the notebook count in the document library crosses 5,000 again, you will start seeing the OneNote API failures and you will need to come back here and take the same remedial steps.

Comments

Anonymous
September 13, 2016
Hi PareshThanks for providing the diagnostic tool and pointing out the issue. As my app has to deal with thousands of notebooks (each with ~10 sections) I've been acutely aware of this issue and already move notebooks that are not actively being worked on out of the Documents folder. Is there any scope for providing a way of just flagging a notebook that should be removed from the "active list" to reduce the resource usage without having to go to all the trouble of moving documents back and forth? Even if it was a file naming convention such a certain character sequence that allows SharePoint to skip a certain notebook folder?By the way loving the Onenote api, any chance it might make it out of beta any time soon?Cheers Steve
- Anonymous
  September 14, 2016
  HI SteveI am glad that you are enjoying using OneNote API. This is great suggestion but it would not work for this scenario. Inactive notebook would work as secondary filter. Once we mark a notebook inactive, we need to ask SharePoint “Give me all notebooks but not inactive”. SharePoint still need to filter notebooks as primary filter and when its scan goes beyond 5000 it will throw throttling exception to OneNote API layer. We have couple of solutions but that would make API slow and hard to use. Considering this fact, the work around is simple. I am also updating the tool to allow users to move notebooks across document library.
  - Anonymous
    September 13, 2017
    You still have an unresolved issue here. My import fails with NO files or folders in SharePoint.See thread at:https://answers.microsoft.com/en-us/msoffice/forum/msoffice_onedrivefb-mso_other-mso_o365b/evernote-to-onenote-import-fails/cf762ba1-00b4-43bc-9856-ec1fba07449eNo one seems to have an answer.Thx
Anonymous
October 03, 2016
What are you going to do about your own applications using this API, such as “Save emails to OneNote” (aka me@onenote.com)? Because of the 20,000 items limitation I can’t even set up this first-party service.
- Anonymous
  October 11, 2016
  Hi JCThis problem is not specific to service but to the storage. SharePoint team has published good articles on how to manage large list/library. I have pulled information from that which is specific to OneNote service. User falls into above categories may have similar issues with our feature like “Save Emails to OneNote”, Web Clippers etc. User should be running above diagnostics against personal site to solve the issue.
  - Anonymous
    April 29, 2017
    Well,they just published something more:https://blogs.office.com/2015/12/16/onedrive-for-business-update-on-storage-plans-and-next-generation-sync-client/...
  - Anonymous
    July 12, 2017
    Hello I don't have sharepoint, nor do I have a lot of notebooks (only one notebook with less than 10 sections. Still webclipper not working. ...I am thinking of uninstalling the webclipper
Anonymous
February 17, 2017
None of these cases apply to me yet I am still having the issue where it will not load the list of notebooks. Here is the output from the tool. Any ideas?List: DocumentsItems Count: 11082Notebooks Count:6Folders Count:2683Sections Count:63Content Type ID:TrueFile Type:TrueHTML File Type:True------------------------------------List: Form TemplatesItems Count: 0Notebooks Count:0Folders Count:0Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:FalseContent Type ID:False------------------------------------List: Style LibraryItems Count: 5Notebooks Count:0Folders Count:1Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False
- Anonymous
  March 12, 2018
  Same issue for me. My numbers are all still below the thresholds, yet I still get error 10008 when tryign to import. Want people to switch from Evernote to OneNote? This is not how to do it.
Anonymous
March 11, 2017
The comment has been removed
Anonymous
April 03, 2017
Please lift the limit on the number of notebooks/pages and folders in regards to being able to use the one note clipper and also saving emails from outlook to one note. I house several notebooks and documents on my business sharepoint site that I must keep active and intact. Moving the items to another person's notebook or location is not an option and I am sure this is the case for many others that utilize these Microsoft products in the workplace.
Anonymous
April 17, 2017
Still unable to load notebooks in OneNote Web Clipper and getting error message about a list limit being exceeded in Sharepoint. Running diagnostic tool on all lists shows:List: Form TemplatesItems Count: 0Notebooks Count:0Folders Count:0Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:FalseContent Type ID:False------------------------------------List: Incoming FaxesItems Count: 0Notebooks Count:0Folders Count:0Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: Legal ReferenceItems Count: 3Notebooks Count:0Folders Count:0Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: Master FormsItems Count: 1224Notebooks Count:0Folders Count:101Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: OneNote NotebooksItems Count: 203Notebooks Count:4Folders Count:33Sections Count:117Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: Permission Test LibraryItems Count: 2Notebooks Count:0Folders Count:0Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: Shared DocumentsItems Count: 2Notebooks Count:0Folders Count:0Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: Site AssetsItems Count: 3Notebooks Count:0Folders Count:2Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: Site Collection DocumentsItems Count: 0Notebooks Count:0Folders Count:0Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:False------------------------------------List: Style LibraryItems Count: 496Notebooks Count:0Folders Count:163Sections Count:0Content Type ID:FalseFile Type:FalseHTML File Type:FalseI do not seem to be close to any list limits but tool is still not working. Please advise.
Anonymous
April 17, 2017
The 20,000 items limit was removed on OneDrive 2 years ago (https://blogs.office.com/2015/12/16/onedrive-for-business-update-on-storage-plans-and-next-generation-sync-client/), it is your API that is not working properly.So what?
- Anonymous
  April 17, 2017
  In add, the tool doesn't show anything. It work a few seconds, than show again the regular windows without any data.
- Anonymous
  May 27, 2017
  If the OneNote API doesn't allow more than 20,000 items, then the limit was NOT removed from OneNote.
  - Anonymous
    November 13, 2017
    Dear friend, thank you for your answer.Sadly the error message doesn't talk about "OneNote", it talk about OneDriver ("One or more of the document libraries on the user or group's OneDrive contains more than 20,000 items and cannot be indexed for querying using the API. Please follow the link below for instructions on how to remedy this situation.",)However, I have no note in OneNote, it is empty.In any case I think some developer "losed himsef in a glass of water" (italian sayng) becouse at least imagine a busines company, as small as you want, that work with less then 20000 files is a very strange idea.To think to move files to other storage as suggested is a worse cure than evil because it means to change workflows in the company, so it is far better to understand that flow doesn't work and do not use it.
    - Anonymous
      November 15, 2017
      Please forgive me for a "losed" I never intended to use, I was very distracted...of course it was "lost"
Anonymous
May 06, 2017
Please Satya Nadella, Get Microsoft's act together. It is silly to have item limitations and limitations on the path length on OneDrive. Other Cloud Drives don't have these limitations.
Anonymous
May 24, 2017
Ugh, this issue is ridiculous. What century are we in? :(
Anonymous
May 27, 2017
This is ridiculous. OneNote CHOKES and is UNUSABLE if your OneDrive contains more than 20,000 items. Come on. Are you kidding me?Evernote handles an unlimited number of notes and items. This is why I still have to use Evernote.OneNote is too limited.
Anonymous
June 02, 2017
The comment has been removed
Anonymous
June 14, 2017
I find it sort of funny that I can't log in -here- using my MS Hosted (Business) credentials to leave a comment. Just want to chime in on the hilarious problem of 20,000 items. I just migrated a database over to my OneDrive that consists of about 180,000 files/folders give or take. Because of this I can't use the web clipper for OneNote. Even better- my one note classroom notebooks HAVE DISAPPEARED. I have grades due in a few hours and I'm guessing MS really couldn't give a flying fig. Our IT people said that MS had started changing things w/ our hosted services without notifying them (I see a new 'classroom' app that is 'in process of being installed for the last 3 days.)And to think what a place like I'm at pays for this suite of services.
Anonymous
July 04, 2017
Dear OneNote team,there was a company decision to move over to Office 365 with all it's opportunities and I appreciated the standardization of tools. Nevertheless recognizing now the limitations in using the OneNote, I'm not sure whether I'm still standing behind this decision. For a company like Microsoft, who wants to be a leader in "Digitalization" it's a shame that limits like this hinders me, to use the full capabilities of OneNote and WebClipper. Why can companies like Evernote can handle this without limitation and Microsoft still struggle! If you want to be seen as leader then you should act accordingly!I would appreciate your feedback - not in wording but in changing/taking away the existing limits.
Anonymous
July 11, 2017
Hi, I'm having this issue every time I try to save or clip anything. I do not have an active sharepoint site whatsoever, and I only have 3 OneNote Notebooks, which in total, account for 8 notes... so none of these remedies seem to fit the issues I'm having whatsoever. I am I 1-man Office 365 account, and my file limits are thousand and thousands of files under the limits mentioned above.
Anonymous
August 08, 2017
Getting the dreaded 10008 import error in a notebook with 3500 items, and my SharePoint library is completely empty. I do have tens of thousands of files in the new client OneDrive for Business library but the documentation clearly implies these are separate limits. Any help appreciated and happy to provide any more info I can. Thanks.
Anonymous
August 13, 2017
tnx i used it
Anonymous
August 13, 2017
oh thats it
Anonymous
October 31, 2017
+1.Onedrive gives us 1To and we're supposed to stay below 20,000 files to be able to use Onenote clipper? It's ridiculous.
Anonymous
November 08, 2017
With IE 11 the add-in still works as expected even with +5k items in my OD4B and a huge Onenote repository. No issue at all and even searching works great!So why do we fall back to a more worse expirience within a new browser that more likely should improve the feture rather than make it more worse than before.What is the caveat to have it for Egde the same as in IE11 instead of using an API that has all the limitations we removed already on almost everywhere in SPO? I don't get this.
Anonymous
December 05, 2017
I had been using the onenote clipper, slowly moving away from Evernote, when the problem of too many items came up. I downloaded the tool, reduced my items to 12,533 and still can't make the clipper work. Tried on multiple OS (win 7, win 10, linux) and multiple browsers (chrome and firefox) but no luck. Here is what the notebook API Diagnostics reports for my Onedrive:Items Count: 12533Notebooks Count:0Folders Count:1Sections Count:0Content Type ID:TrueFile Type:TrueHTML File Type:TrueI must be missing something . . .but what?
Anonymous
December 11, 2017
This is an idiotic limitation. I was storing all of my work files/folders on Onedrive as opposed to alternative options (dropbox, google drive, etc.) since the sync capability was shockingly better-behaved with Onedrive. I should have known there was a catch, and apparently I found it. Now I have to re-arrange my files to get the OneNote clipper and Outlook options to work, what a waste of time.
Anonymous
December 26, 2017
No more onenote for me. I don't have time for this.
Anonymous
January 02, 2018
Please get this fixed. It's been way too long and there are too many people with this same problem. It makes no sense to pay for 1 TB of storage space only to be severely limited by Web Clipper
- Anonymous
  February 16, 2018
  It's disappointing to have limits placed on OneDrive file numbers simply to get OneNote web clipper to work with a handful of notebooks.
Anonymous
March 14, 2018
I get the following error;System.Xml.XmlException: For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method.
Anonymous
April 10, 2018
At the end you got me to use OneNote instead of Evernote - and I am a convinced user of Evernote. Now, I want to move my account to OneNote and you tell me that's too much storage to import (Code 10008) and I should manually divide my existing Evernote notebooks to several accounts? No! The 90s are over and Microsoft is not alone in the universe anymore. So, I will stay with Evernote.
Anonymous
November 21, 2018
I know, more than two years later. But I had the same problem getting around only 100 Notes from Evernote to OneNote via the import tool. It finally worked when I unchecked "Use Evernote tags to organize content...". This was reproducable, choosing this option always resulted in Code 10008.
Anonymous
February 03, 2019
The comment has been removed
- Anonymous
  February 13, 2019
  The comment has been removed
  - Anonymous
    April 18, 2019
    Yes. This issue needs addressed ASAP. I should not have to delete files from my onedrive in order to use the onenote web clipper. If the limitation is not present in the modern interface, then it needs to be fixed in the API too.
    - Anonymous
      April 19, 2019
      Hilariously, I decided to visit this page again to see if there has been any progress here and, indeed, a comment posted here -just yesterday- about how absurd this problem is and remains. Ugh. I'm amazed this is still an issue.
Anonymous
May 10, 2019
An example of how Microsoft's UX initiative has not yet penetrated every corner of the company. How can one offer an end user of the OneNote Web Clipper this expert-only page after an error? I'm using my business O365 account to search the web and I won't delete individual items from my OneNote files under any circumstances. How can it happen at a time when a data volume of gigabytes or even terabytes is part of an O365 contract that I can no longer save a single URL (!) due to a limited number of anything? As an end user, I simply do not want to care about the (obviously not scalable) inner workings of O365. This is so grotesque that I can hardly believe it.
Anonymous
May 22, 2019
How can you possibly call 20K items a very large document library? 20,000 is small for any enterprise, and only 0.67% of the 30,000,000 items (files + folders) that SharePoint Online can handle! This kills the use of OneNote stone dead in any Teams SPO document library with more than 20K items. It may be by design, but the design is outdated and no longer fit for purpose, please revisit!
Anonymous
June 11, 2019
The comment has been removed

Share via