Very slow upload to Archive blob (70MBps->0.004MBps)

Arie Youlus 21 Reputation points
2024-01-01T08:23:58.0733333+00:00

I am currently in the process of transferring several terabytes of video data to my Azure Blob Storage account, specifically using the Archive tier for the blobs.

To facilitate the upload, I am employing azcopy, and the machine dedicated to this task operates with a standalone internet connection, with no other devices connected. The connection speed is consistently at 70 megabytes per second.

Over the past few weeks, the azcopy run has been indicating a throughput of 60-70, but the actual average upload speed is approximately 10-15 gigabytes per hour, equivalent to 0.0031-0.0041 megabytes per second.

We have thoroughly examined the machine for any potential issues, but everything appears to be functioning correctly. While I acknowledge that Azure may impose speed limits to prevent congestion, the significant disparity between the anticipated and observed upload speeds is a cause for concern.

I would greatly appreciate your assistance in resolving this matter.

Thank you

Arie

Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
240 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,871 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,575 questions
Azure
Azure
A cloud computing platform and infrastructure for building, deploying and managing applications and services through a worldwide network of Microsoft-managed datacenters.
1,056 questions

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Vinodh247 12,746 Reputation points
    2024-01-01T10:44:13.5266667+00:00

    Thanks for reaching out to Microsoft Q&A.

    If you have not optimized your azcopy for larger workloads, pls go through the below link... I hope this will give you some insights

    https://video2.skills-academy.com/en-us/azure/storage/common/storage-use-azcopy-optimize

    https://video2.skills-academy.com/en-us/azure/storage/blobs/scalability-targets#scale-targets-for-blob-storage

    Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

  2. Sumarigo-MSFT 44,906 Reputation points Microsoft Employee
    2024-02-04T05:59:31.9866667+00:00

    @Arie Youlus Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

    Adding more information to the above response
    Have you check the time taken in any other way for uploading such as via the Portal or via storage explorer and have you noticed it to be faster or same speed?

    The latency depends upon your location and the storage location and distance between them as well as your link speed and many other factors. Where are you uploading the data from and where is your storage account located? You can also test upload to Azure storage using this link- https://www.azurespeed.com/Azure/Upload

    There is no throttling on the Azure side, refer to below article. If the network and client machine can handle the traffic then we will send it.

    This article provides an overview of some of the common Azure data transfer solutions. The article also links out to recommended options depending on the network bandwidth in your environment and the size of the data you intend to transfer.
    https://video2.skills-academy.com/en-us/azure/storage/common/storage-choose-data-transfer-solution Azcopy Optimize for large numbers of files : Throughput can decrease when transferring large numbers of files. Each copy operation translates to one or more transactions that must be executed in the storage service. When you are transferring a large number of files, consider the number of transactions that need to be executed and any potential impact those transactions can have if other activities are occurring in the storage account at the same time. To maximize performance, you can reduce the size of each job by limiting the number of files that are copied in a single job. For download and upload operations, increase concurrency as needed, decrease log activity, and turn off features that incur high performance costs.

    • There could be several reasons for this issue, such as network congestion, bandwidth limitations, or throttling by Azure. Here are some steps you can take to troubleshoot the issue:
    • Check the network connection: Make sure that the network connection is stable and there are no issues with the internet connection. You can try uploading the files from a different network or computer to see if the issue persists.
    • Check the bandwidth limitations: Make sure that there are no bandwidth limitations imposed by your internet service provider or by Azure. You can check the Azure documentation for more information on the bandwidth limitations for Azure Blob Storage.
    • Check the throttling: Make sure that there is no throttling by Azure. Azure may impose throttling to prevent congestion and ensure fair usage of resources. You can check the Azure documentation for more information on the throttling policies for Azure Blob Storage.
    • Check the azcopy settings: Make sure that the azcopy settings are optimized for the upload. You can check the azcopy documentation for more information on the recommended settings for large-scale data transfers.
    • Check the Azure Blob Storage settings: Make sure that the Azure Blob Storage settings are optimized for the upload. You can check the Azure documentation for more information on the recommended settings for large-scale data transfers.

    Optimize the performance of AzCopy with Azure Storage

    Please let us know if you have any further queries. I’m happy to assist you further.     

    ---Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments