AzCopy – Uploading/Downloading files for Windows Azure Blobs

  • Artigo

One of the frequent requests we receive is for a simple way to upload or download files between Windows Azure Blob Storage and their local file system. We’re pleased to release AzCopy (Beta Version), which is a command line utility which allows Windows Azure Storage customers to do just that. The utility is designed to simplify the task of transferring data in to and out of a Windows Azure Storage account. Customers can use this as a standalone tool or incorporate this utility in an existing application. You can always download latest version from aka.ms/AzCopy.

The command is analogous to other Microsoft file copy utilities like robocopy that you may already be familiar with. Below is the syntax:

AzCopy <Source> <Destination> [filepattern [filepattern…]] [Options]

In this post we highlight AzCopy’s features without going into implementation details. The help command (AzCopy /?) lists and briefly describes all available system commands and parameters.

Key Features:

  • Efficient and Flexible: AzCopy allows users to selectively copy the data. By using recursive mode, customers can copy nested directories of files. Users can specify file pattern – wild cards in Windows file system or prefix in the blob storage to identify source files that are candidates for copying. In addition, users can set an option to copy files which have “Archive” attribute set. When copying a large number of files, and if there is a copy failure due to network or other issues, the re-startable option can resume the copy process from where it left off (the files which have not been copied). Using re-startable mode, you will not need to re-copy files which were successful in the previous attempt.
  • Support for Windows Azure Storage Data Types: AzCopy provides options for customers to specify the destination data in a storage account as a block blob or a page blob. Default is set to block blobs, as this is best suited for nearly all files. When using the page blob option, the blob will be zero-padded to a 512-byte boundary.
  • Naming Rules: URI format (http or https) is used to specify the blob storage path and NTFS Windows file folder path is used for Windows file system. Since some of the blob names cannot be directly translated into Windows file system, AzCopy translates into Windows file system names using its own rules. Here are some rules that we follow in naming translations:
    • Blobs that conflict with Windows special file names will be renamed using the following rules: “.” => “dot”; “..” => “dotdot”; “/” => “slash”; etc. As with other conflict resolution rules, if a conflict occurs on any of these names the string “(n)” will be added to the conflicting file or blob name such that the conflict is resolved.
    • The Windows file system namespace is case insensitive (but case retentive), and the Windows Azure blob namespace case sensitive, the following rules apply:
      • Blobs in the blob namespace are created with the default case for the file name.
      • Files are created with the default case of the blob namespace.
      • If a case-conflict occurs while copying from the blob-namespace to the file-namespace, we will append the string “(n)” to the conflicting file or blob names.
  • Logging: Users can run AzCopy in a verbose mode, which displays lists of files and directories processed, and render the list of files that the utility failed to copy. AzCopy also displays the progress indication for each file while running in verbose mode.

Examples:

Example 1: Copy a directory of locally accessible files to blob storage container in a recursive mode.

AzCopy C:\blob-data https://myaccount.blob.core.windows.net/mycontainer/ /destkey:key /S

The above command will copy all files from the “c:\blob-data” directory and all subdirectories as block blobs to the container named “mycontainer” in storage account “myaccount”. “Blob-data” folder contains the following files and one subdirectory named “subfolder1”;

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\train1.docx

C:\blob-data\subfolder1\car_sub1.docx

C:\blob-data\subfolder1\car_sub2.docx

After the copy operation, “mycontainer” blob container will contain the following blobs:

car1.docx

car2.docx

car3.docx

train1.docx

subfolder1/car_sub1.docx

subfolder1/car_sub2.docx

 

If we do not use recursive mode (Copying without the “/S” option), the “mycontainer” blob container would only contain the following files under “blob-data” folder and would ignore files under the “subfolder1” folder.

car1.docx

car2.docx

car3.docx

train1.docx

Example 2: Recursively copy a set of blobs from a blob storage to a locally accessible directory in both verbose and recursive modes.

AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data /sourceKey:key /S /V

The command will copy all blobs under the “mycontainer” blob container in account “myaccount” to the “c:\blob-data” directory in both verbose and recursive modes.

“mycontainer” blob container contains the following files:

car1.docx

car2.docx

car3.docx

train1.docx

subfolder1/car_sub1.docx

subfolder1/car_sub2.docx

 

Since we are using the verbose mode, the tool will display the following output which contains the file transfer status of each of the file in addition to the transfer summary. By default, the tool will only display the transfer summary:

Finished Transfer: car1.docx

Finished Transfer: car2.docx

Finished Transfer: car3.docx

Finished Transfer: train1.docx

Finished Transfer: subfolder1/car_sub1.docx

Finished Transfer: subfolder1/car_sub2.docx

Transfer summary:

-----------------

Total files transferred: 6

Transfer successfully: 6

Transfer failed: 0

After the copy operation, c:\blob-data folder will contain the files listed below:

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\train1.docx

C:\blob-data\subfolder1\car_sub1.docx

C:\blob-data\subfolder1\car_sub2.docx

 

Let’s try a slightly different scenario by copying the blobs which start with “subfolder1\” by using the following command:

AzCopy https://myaccount.blob.core.windows.net/mycontainer/subfolder1 c:\blob-data /sourceKey:key /S /V

The above command will only copy blobs which begin with “subfolder1/”, and thus the tool will only copy “subfolder1/car_sub1.docx” and “subfolder1/car_sub2.docx” blobs to “c:\blob-data\” folder. After the copy operation, “C:\blob-data” will contain the following files:

C:\blob-data\car_sub1.docx

C:\blob-data\car_sub2.docx

Example 3: Copy a directory of locally accessible files to a blob account in re-startable mode

AzCopy c:\blob-data https://myaccount.blob.core.windows.net/mycontainer /destkey:key /Z:restart.log /S

Restart.log, a journal file, will be used to maintain a record of the status of the copy operation to allow the operation to restart if interrupted. If there is no text file specifies along with the re-startable mode parameter, the journal file will default to “azcopy.log” in the current working directory.

For instance, “C:\blob-data” folder contains the five large files with each of the file size greater than 100 MB.

C:\blob-data\car.docx

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\car4.docx

 

When running with restart option, AzCopy allows you to restart the process in the case of failure. If the failure occurred while copying “car.docx”, AzCopy will resume the copy from the part of “car.docx” which has not been copied. If the copy occurred after “car.docx” was successfully copied, AzCopy will resume the copy operation from one of the remaining four files which have yet to be copied.

Example 4: Select number of files in a storage blob container using a file pattern and copy them to a locally accessible directory.

AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data car /sourceKey:key /Z /S

“mycontainer” contains the following files:

car1.docx

car2.docx

car3.docx

train.docx

carfolder/car_sub1.docx

carfolder/train_sub2.docx

subfolder1/car_sub1.docx

subfolder1/car_sub2.docx

 

After copy operation, “c:\blob-data” will contain the files listed below. Since the file pattern with the prefix of “car” was specified, the copy operation copies only the file with the prefix of “car”. Note that this prefix is applied to the blob, if it’s directly in the “mycontainer” container, or to the subdirectory name.

C:\blob-data\car1.docx

C:\blob-data\car2.docx

C:\blob-data\car3.docx

C:\blob-data\carfolder\car_sub1.docx

C:\blob-data\carfolder\train_sub2.docx

 

Performance

Within a Windows Azure datacenter (i.e., between a compute instance and a storage account within the same DC), users should be able to achieve 50MB/s upload and download speed for uploading and downloading large amounts of data using an extra-large compute instance. Transfers to and from a Windows Azure datacenter will be constrained by the bandwidth available to AzCopy.

Known Issues

  • When copying without /XN (Exclude Newer) and /XO (Exclude Older), the tool only compares the names of source and target files before copying. Therefore, users will be prompted whether to overwrite the target files although source and target files are identical.
  • When using /XN and /XO, note that your local system time and the time stored in the storage service will vary slightly. So if the blob and local file were modified at nearly the same time, this comparison may not filter correctly.
  • When copying a file to a page blob with the re-startable option and if there is a copy operation failure in the middle of copying, the tool will restart the copy process from the beginning of the file. This issue does not apply to copying a file to a block blob.
  • When copying blobs to local %systemdrive%, the tool will not prompt the confirm message to overwrite or not to overwrite the files if there are existing files with the same name.
  • If there are two blobs named “a” and “a/b” under a storage container, copying the blobs under that container with /S will fail. Windows will not allow the creation of folder name “a” and file name “a” under the same folder.

 

Aung Oo
Matthew Hendel

Windows Azure Storage Team

Comments

  • Anonymous
    December 10, 2012
    Nice. Any plans to support the following?
  1. An option similar to robocopy /MIR which would also delete and not just add/update files at the destination.
  2. File comparison by MD5 instead or in addition to file dates to reliably identify modified files.
  3. GZIP for selected file types when uploading to Azure? Thanks!

  • Anonymous
    December 10, 2012
    Thanks for your feature request.  We will take them into consideration for future releases.

  • Anonymous
    December 19, 2012
    The comment has been removed

  • Anonymous
    December 19, 2012
    The comment has been removed

  • Anonymous
    December 19, 2012
    If Source and Dest both are containers, doesn't work.

  • Anonymous
    January 11, 2013
    Please also include a link for the source code... all I see is the binary.  I want to understand why the HPC library is used/needed

  • Anonymous
    January 13, 2013
    Hi ChrisL, We only share binary with this version and working towards sharing the code in later releases.  The current version leverages HPC library to copy blobs to the blob storage account and thus it is needed.

  • Anonymous
    February 15, 2013
    Thanks...very useful. What options are available for folks operating behind ISA Client. I'm receiving "The remote server returned an error: (407) Proxy Authentication Required. What method would I use to control proxy settings, such as... <system.net> <defaultProxy enabled="true" useDefaultCredentials="true">     <proxy usesystemdefault="True" /> </defaultProxy> </system.net>

  • Anonymous
    February 15, 2013
    @KevinTag:  Please refer to this thread in the networking forum: social.msdn.microsoft.com/.../06a45fcf-56d9-4dda-a9b3-0d46addc357f.  Thanks!

  • Anonymous
    February 16, 2013
    Is there any way to retain the modified date information when copying?

  • Anonymous
    February 16, 2013
    Hi Alan, This is currently not supported, however we will take this as a feature request. Thanks, Jean

  • Anonymous
    March 11, 2013
    You can also use the Blob Transfer Utility to download and upload all your blob files. It's a GUI tool to handle thousands of blob transfers in a effective way. Binaries and source code, here: http://bit.ly/blobtransfer

  • Anonymous
    March 13, 2013
    I'm trying to upload data to a HDInsight cluster. Using azcopy was recommended but getting an error: "Error parsing destination location". I provide my cluster id found on my HDInsight home page.   Can you provide an example for HDInsight use? thanks.

  • Anonymous
    April 08, 2013
    AzCopy supports upload data from local to Azure Storage, however HDInsight Cluster is not a valid destination for AzCopy yet. We will take this into consideration for later release. -Jason TANG (Microsoft)

  • Anonymous
    June 04, 2013
    Can I set MIME image/jpg for uploading image files?

  • Anonymous
    October 17, 2013
    Thanks a lot....a great tool...Please share your GitHUB URL..so that i can mention it in my blog Sandesh Daddi http://www.sanshark.com

  • Anonymous
    November 12, 2013
    The comment has been removed

  • Anonymous
    December 19, 2013
    The comment has been removed

  • Anonymous
    January 26, 2014
    Nice post. .. Any help on DELETE, RENAME, COPY etc. operations using AzCopy ?

  • Anonymous
    February 19, 2014
    The comment has been removed

  • Anonymous
    March 25, 2014
    The comment has been removed

  • Anonymous
    March 25, 2014
    Hi Mark, We will consider to open source AzCopy in the future, but do not have a timeline yet. Regards. Zhiming

  • Anonymous
    April 07, 2014
    Any plan to see AzCopy as a .NET library? It was so cool to have this speed in our custom code!

  • Anonymous
    April 07, 2014
    Hi Andrea, Thank you for the feedback, we’ll add that to our feature request list. Zhiming

  • Anonymous
    April 14, 2014
    Hi, Nice Article. Is there a option to change the source file name, when I copy from Blob to application server?

  • Anonymous
    April 15, 2014
    Hi Vivek, Thanks for the feedback. AzCopy does not support to change blob name when copy from source to destination. We will add this to the list of feature requests. Regards. Zhiming

  • Anonymous
    May 07, 2014
    Are there any usage restrictions for AzCopy?  Is it under any open source license? Thanks, Mark

  • Anonymous
    May 07, 2014
    The comment has been removed

  • Anonymous
    May 08, 2014
    Is there a way to use AzCopy to transfer files from on premises server to VM in Azure.  So far I've gotten best performance using AzCopy to transfer files from on premise server to Azure BLOB.

  • Anonymous
    May 08, 2014
    Hi NK, AzCopy does not support transferring file to and from Azure-VM’s
    directly. If you have a large number of files or some big files to copy you can
    first use AzCopy on premise to copy the files to the Blob Storage Service, then
    TDP into the VM and copy the files from the Blob Service to the VM. If you only
    have a small number of files you can also just RDP into the VM and copy &
    paste the files using the file explorer. Samarth

  • Anonymous
    June 16, 2014
    When we upload the files using azcopy from local to azure blob the Content-Type is application/octet-stream but we would need to be able to specify by example text/javascript. AzCopy has great speed but because of this Content-Type issue i m unable to implement it  as our JS & CSS files are not reflecting properly in our web application

  • Anonymous
    June 16, 2014
    Hi Sameer, Thanks for your feedback, we will put ‘set content-type’ on our requested feature list. Zhiming

  • Anonymous
    June 16, 2014
    Hi Zhiming, To make it more clear below is my observation of blob attributes using azcopy

  1. While Uploading a new file Content-Type is application/octet-stream
  2. While overriding a existing file Content-Type is retained as per the previous blob e.g if file was gzipped then even after uploading a fresh file from azcopy it keeps content-encoding as gzip but actually file is uncompressed or un-gzipped . Expectation : blob attributes should get set according to local or source file . e.g. for js ="text/javascript" for css ="text/css" etc

  • Anonymous
    June 17, 2014
    Thanks Sameer, We will take your suggestion into consideration when implementing this feature in the future. Zhiming

  • Anonymous
    July 29, 2014
    I want to upload 500MB of data from local drive to blob storage, when I tried this tool with following options "/S /Y /MOV /NC:5 /V /z:upload.log". it took a long time to complete the operation. Is there a faster set of options to use?

  • Anonymous
    July 29, 2014
    Hi Pydi, You may want to remove the option /NC:5 which control the number of concurrent operations, then the default concurrent operations' count will be CORE * 8, and if you do not want to delete your local files after uploading, you may also remove the option /mov. And actually the uploading speed is not only decided by the options you use, but mainly affected by your uploading environment, like network, disk, CPU etc. Zhiming

  • Anonymous
    November 27, 2014
    For avoiding upload or replace what is uploaded running for first time AzCopy, AzCopy ask for replace for each file? Setting /XO parameter helps me to avoid that?

  • Anonymous
    November 27, 2014
    Hi, Néstor Yes, for any conflict between source and destination, AzCopy will prompt asking for overwrite, unless /y is specified which means overwrite all. Usually /xo should help to exclude the uploaded ones. By definition: /XO                           Excludes an older source resource. The resource                              will not be copied if the source resource is                              older than destination. LastModifiedTime of uploaded blob is the time of last uploading, so usually it should be newer then the local source file's LastModifiedTime (unless changed intently) and /xo will exclude it from source. Zac

  • Anonymous
    March 03, 2015
    I am using system with IE has proxy settings, I realize azcopy will auto-pick up the settings from IE , the result is all traffic will go through the proxy server. Is there a switch to ask azcopy do not pickup proxy settings from IE to archive better transfer performance. Or in some case, I want to specify a different proxy server in order to archive reliable network path to storage account Can we add a new proxy switch. /proxy:default or it is not specified (the current logical pickup settings from AutoProxy /proxy:none (no proxy ) /proxy:myproxy.abc.com:1234 (use proxy user specified)              /proxyuser:username   (only used if proxy need authentication and default security context does not work)              /proxypassword:password (only used if proxy need authentication and default security context does not work)

  • Anonymous
    March 16, 2015
    Hi Qing, Please try to add azcopy.exe.config with the following content to your AzCopy installation folder. • Disable proxy used in IE: <configuration>  <system.net>    <defaultProxy enabled=”false” />  </system.net> </configuration> • Customize proxy address: <configuration>  <system.net>    <defaultProxy>      <proxy        proxyaddress="http://127.0.0.1:8080"        bypassonlocal="true"      />    </defaultProxy>  </system.net> </configuration> Please find more details at msdn.microsoft.com/.../aa903360(v=vs.71).aspx that shall solve most of your questions. Zhiming

  • Anonymous
    May 04, 2015
    HI Noppes, There is no constraint of how big the file AzCopy can copy, but please note that the maximum size of block blob is around 200GB, and page blob is 1TB. Zhiming