Title: Seeking Guidance on Syncing/Migrating SharePoint Data to Azure Blob Storage

Shubham Taur 20 Reputation points
2024-02-08T06:48:44.65+00:00

Hello Microsoft Community, I'm currently exploring options for syncing or migrating data from SharePoint to Azure Blob Storage and would greatly appreciate some guidance on the best practices and recommended approaches. Specifically, I'm interested in understanding:

  1. What are the preferred methods or tools for syncing or migrating SharePoint data to Azure Blob Storage?
  2. Are there any built-in features or services within Azure or SharePoint that streamline this process?
  3. How can I ensure data integrity and maintain permissions/access controls during the sync/migration process?
  4. Are there any considerations or potential challenges I should be aware of before initiating the sync/migration?
  5. Are there any best practices for optimizing storage costs or managing data lifecycle post-migration within Azure Blob Storage? Any insights, experiences, or resources shared would be immensely helpful as I navigate this endeavor. Thank you in advance for your assistance! Feel free to adjust the question to better suit your specific needs or preferences! Warm regards, Shubham Taur
SharePoint Server
SharePoint Server
A family of Microsoft on-premises document management and storage systems.
2,287 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,578 questions
SharePoint Server Development
SharePoint Server Development
SharePoint Server: A family of Microsoft on-premises document management and storage systems.Development: The process of researching, productizing, and refining new or existing technologies.
1,594 questions
SharePoint Server Management
SharePoint Server Management
SharePoint Server: A family of Microsoft on-premises document management and storage systems.Management: The act or process of organizing, handling, directing or controlling something.
2,881 questions
Accepted answer
  1. Nehruji R 4,216 Reputation points Microsoft Vendor
    2024-02-09T08:20:58.4966667+00:00

    Hello Shubham Taur,

    Greetings! Welcome to Microsoft Q&A Platform.

    Adding to above response, I would like to provide some more ways to move the data from SharePoint & OneDrive to Azure Blob storage by below following methods.

    1.To transfer files from SharePoint to Azure Blob Storage, there are several methods you can consider, as outlined in Microsoft's documentation:

    You can use Microsoft Power Automate: You can use Power Automate for copying files from a SharePoint folder to an Azure Blob folder, though this might have limitations. Power Automate has a maximum file size limit for individual files. This limit can vary based on the plan you are using, but typically, it's around 100 MB to 250 MB per file also you can create a automated process for transferring data at several intervals.

    refer - https://video2.skills-academy.com/en-us/azure/logic-apps/logic-apps-limits-and-config?tabs=consumption#workflow-definition-limits

    Other reference that might help you: https://sharepains.com/2022/11/15/copy-large-files-sharepoint-azure-blob/

    Also, for automatically transfer data from SharePoint or OneDrive to Azure Blob Storage at specific time intervals.

    You have to create an automated flow that moves files from a specific SharePoint folder to Azure Blob storage after a set period of time. To maintain the folder structure, consider the following approach:

    • When users place a folder within a specific area in SharePoint, use a trigger (e.g., when a new folder is created) to initiate the flow.
    • Inside the flow, use the Azure Blob Storage connector to create the corresponding folder structure in the Azure Blob container.
    • Then, move all the files from the SharePoint folder to the corresponding Azure Blob folder using the Copy File action.
    • This way, both the folder and its files will be maintained in Azure Blob storage1.

    For transferring files from OneDrive to Blob Storage - Microsoft Power Automate

    2.You can also use Azure Data Factory: It is a cloud-based data integration service that allows you to create data pipelines that move and transform data between various sources and destinations, including SharePoint and Azure Blob Storage. You can create a pipeline that retrieves files from SharePoint using the SharePoint connector and then uses the Azure Blob Storage connector to upload the files to Azure Blob Storage. This approach requires some configuration and coding but provides more flexibility and scalability than Logic Apps.

    If there are extremely large files to be copied from that SharePoint library. SharePoint has a default limit of 100 MB buffer size, and the Get File Content action doesn’t natively support chunking.

    You can refer this article to use a binary dataset if you just want to copy the full file rather than read the data. https://datasavvy.me/2021/12/07/copying-large-files-from-sharepoint-online/

    Also check Azure Data Box: It is a physical device provided by Microsoft for offline data transfer. This method is suitable for large data sets and can significantly reduce the time it takes to transfer the data.

    If your files are larger than 250 MB you might consider using other options, like Azure Storage Mover refer-https://video2.skills-academy.com/en-us/azure/storage-mover/service-overview , or can use Date pipelines, or scripted transfer with Azure Automation Account, Maybe Data Box.

    Once the data is moved to Blob storage you can configure the backup using Azure Backup - refer https://video2.skills-academy.com/en-us/azure/backup/blob-backup-configure-manage?tabs=operational-backup for more information.

    Hope this answer helps. Please let us know if you have any further queries. I’m happy to assist you further.

    1 person found this answer helpful.

2 additional answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Azar 21,230 Reputation points MVP
    2024-02-08T11:35:11.1533333+00:00

    Hi there,

    Shubham Taur

    Thats a great question and thanks for using QandA platform.

    let me drop you a few best practices and recommended approaches to consider.

    for Preferred Methods or Tools, You can use Azure Data Factory, which provides a robust and scalable platform for data integration and migration tasks. Another option is to leverage SharePoint Online's built-in migration capabilities or third-party migration tools that support SharePoint-to-Azure Blob Storage migration.

    Azure Blob Storage offers several features that can streamline the migration process, such as Azure Data Factory's copy activity or Azure Storage Explorer for bulk uploads

    It's crucial to mke sure data integrity and maintain permissions during the migration process. Azure Data Factory allows you to preserve metadata and permissions during data movement.

    some of the challenges I would say are to be aware of include handling large volumes of data, managing metadata mappings between SharePoint and Azure Blob Storage, and dealing with any customizations or complex data structures in SharePoint. It's essential to thoroughly plan and test your migration strategy to mitigate these challenges.

    If this helps kindly accept the answer thanks much.

  2. Shubham Taur 20 Reputation points
    2024-02-09T12:48:36.09+00:00

    Hii Nehruji R,Can you please share them with me to process Documents?