Very slow when writing lots of files to Blob Storage

Peter Ma 211 Reputation points Microsoft Employee
2021-07-14T00:39:05.303+00:00

I wrote about 1000 small files(~0.5 mb/file) to blob storage from ADF.
I only found 70 files by Azure sotrage explorer after ADF activity finished.

When I refresh, there will be 1 or 2 files added to the folder like the following image show.

My question is, are all files wrote to blob storage at that time when ADF activity finsihed. Because I have another activity to consume these files but I don't know when all the files can be writen to blob storage?

114347-image.png

Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
240 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,575 questions
0 comments
Accepted answer
  1. Sumarigo-MSFT 44,906 Reputation points Microsoft Employee
    2021-07-14T15:31:34.507+00:00

    anonymous user-0048 Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

    There could be several reason, for the performance issue (Network bandwidth, intermittently issue may occur). Can you perform
    https://www.azurespeed.com/Azure/Download

    Which version on Azure Storage explorer are using? Can you try to un-install and re-install the latest version of Storage Explorer.

    For testing purpose you can also use azcopy tool and check the performance

    Storage Explorer currently doesn't have the functionality for users to configure our usage of the network bandwidth. It is something we want to enable in the future as a part of the configuration. There are two workarounds I can think of in addition to using FTP. Storage Explorer has a built-in Experimental feature called AzCopy integration, which dramatically speeds up blob uploading and downloading. You can also use AzCopy at command line which actually gives you the ability to configure different parameters including the bandwidth you want to use during upload/download. Hope this is informational.

    Additional information: There is no throttling on the Azure side below the published 60 MB/s scalability target for a single blob. If the network and client machine can handle the traffic then we will send it.

    Here's a guide for troubleshooting high E2E latency: https://video2.skills-academy.com/en-us/azure/storage/common/storage-monitoring-diagnosing-troubleshooting#metrics-show-high-AverageE2ELatency-and-low-AverageServerLatency

    Kindly let us know if the above helps or you need further assistance on this issue.

    ----------------------------------------------------------------------------------------------------------------------------------------------

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest