AzCopy – Using Cross Account Copy Blob

  • Articolo

You can always download latest version from aka.ms/AzCopy.

New features have been added in this release

  • Support of Cross-Account Copy Blob: AzCopy allows you to copy blobs within same storage accounts or between different storage accounts (visit this blog post for more details on cross account blob copy). This enables you move blobs from one account to another efficiently with respect to cost and time. The data transfer is done by the storage service, thus eliminating the need for you to download each blob from the source and then upload to the destination. You can also use /Z to execute the blob copy in re-startable mode.

  • Added /MOV: This option allows you move files and delete them from source after copying. Assuming you have delete permissions on the source, this option applies regardless of whether the source is Windows Azure Storage or the local file system.

  • Added /NC: This option allows you to specify the concurrent network calls. By default, when you upload files from local computer to Windows Azure Storage, AzCopy will initiate network calls up to eight times the number of cores this local computer had to execute concurrent tasks. For example, if your local computer has four cores, then AzCopy will initiate up to 32 (eight times of 4) network calls at one time. However, if you want to limit the concurrency to throttle local CPU and bandwidth usage, you can specify the maximum concurrent network calls by using /NC. The value specified here is the absolute count and will not be multiplied by the core count. So in the above example, to reduce the concurrent network calls by half, you would specify /NC:16

  • Added /SNAPSHOT: This option allows you transfer blob with snapshots. This is a semantic change, as in AzCopy CTP 1 (Released in October, 2012), it would transfer a blob’s snapshots by default.  However starting from this version, by default AzCopy won’t transfer any snapshots while copying a blob. Only with /SNAPSHOT specified, AzCopy will actually transfer all the snapshots of a blob to destination, however these snapshots will became separate blobs instead of snapshots of original base blob in destination, so each of the blobs will be full charged for (not block reuse between them). The transferred blob snapshots will be renamed in this format: [blob-name] (snapshot-time)[extension].

    For example if readme.txt is a source blob and there are 3 snapshots created for it in source container, then after using /SNAPSHOT there will be 3 more separate blobs created in destination container where their names will looks like

    readme (2013-02-25 080757).txt
    readme (2012-12-23 120657).txt
    readme (2012-09-12 090521).txt

    For the billing impact compare blob snapshots to separate blob, please refer to this blog post Understanding Windows Azure Storage Billing

  • Added /@:response-file: Thisallows you store parameters in a file and they will be processed by AzCopy just as if they had been specified on the command line. Parameters in the response file can be divided as several lines but each single parameter must be in one line (breaking 1 parameter into 2 lines is not supported). AzCopy parses each line as if it is a single command line into a list of parameters and concatenates all the parameter lists into one list, which is treated by AzCopy as from a single command line. Multiple response files can be specified, but nested response file is not supported, instead it will be parsed as a location parameter or file pattern. Escape characters are not supported, except that “” in a quoted string will be parsed as a single quotation mark. Note that /@: after /- does not means a response file, it is treated as a location parameter or file pattern like other parameters

Examples

Here are some examples that illustrate the new features in this release.

Copy all blobs from one container to another container under different storage account

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /S

The above command will copy all blobs from the container named “sourcecontainer” in storage account “sourceaccount” to another container named “destcontainer” in storage account “destaccount”

If you have base blob with snapshots, please add/Snapshot to move all snapshots with base blob to destination, please be noted the blob snapshot will be renamed to this format in destination: [blob-name] (snapshot-time)[extension]

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /S /SNAPSHOT

For example if you have readme.txt with 3 snapshots in source container, then it will be as below in destination container
readme.txt

readme (2013-02-25 080757).txt
readme (2012-12-23 120657).txt
readme (2012-09-12 090521).txt

If you’d like to delete those blobs from the source container when the copy is complete, then just added /MOV as below

AzCopy https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/ https://<destaccount>.blob.core.windows.net/<destcontainer>/  /sourcekey:<key> /destkey:<key> /MOV /S

You can also create a response file to make it easier of running same command again and again. Create a txt file called “myAzCopy.txt” with content below

#URI of Source Container
https://<sourceaccount>.blob.core.windows.net/<sourcecontainer>/
#URI of Destination Container
https://<destaccount>.blob.core.windows.net/<destcontainer>/

Then you can run the command below to transfer files from source container to destination container

AzCopy </@:C:\myAzCopy.txt> /sourcekey:<key> /destkey:<key> /MOV /S

Comments

  • Anonymous
    May 19, 2013
    Hello, Thanks a lot for this amazing tool. Is there any plan to add a timeout switch to enable large file copy (by increasing it). Or, will you release the source code so we can submit contribution to this tool. Best regards, Cyprien

  • Anonymous
    June 24, 2013
    Hello, Does this copy also the Blob metadata? Thanks.

  • Anonymous
    June 25, 2013
    @Ronen - Yes it copies the metadata. It internally uses (async) CopyBlob across storage accounts see msdn.microsoft.com/.../dd894037.aspx for details on other properties that the API sets. @Sandorfr - Timeout switch is a good request. We will add it to the list of feature requests.

  • Anonymous
    August 28, 2013
    Is there a companion command to get the names of the files that are in the source container?

  • Anonymous
    August 28, 2013
    The comment has been removed

  • Anonymous
    September 27, 2013
    I am not sure what I am doing wrong but nothing I try seems to work. It doesn't throw any errors but no matter what I try it always prints the following: Transfer summary:

Total files transferred: 0 Transfer successfully:   0 Transfer failed:         0 I tried copying blobs between storage accounts in the same subscription. I tried copying blobs between storage accounts in different subscriptions. I tried making the blob containers public. I tried blob URLs with and without SSL. I tried specifying a single blob in the source and a container in the destination. I tried specifying containers in the source and destination. I tried with and without trailing slash in the container path. All to no avail. It resolutely produces the above given output. I tried running it with Fiddler on and all I see is 2 blob listing calls for the source and destination accounts with nothing after that. At this point I am ready to give up and write my own thing. Is the source for this tool up on github? I tried searching in the Windows Azure github account and I couldn't find it there!

  • Anonymous
    September 29, 2013
    @Raj - Source of this tool is not yet opened to public. Could you please try to add /S to see whether it resolves your issue? AzCopy sourceaccount.blob.core.windows.net/sourcecontainer destaccount.blob.core.windows.net/destcontainer /SourceKey:****** /DestKey:****** /S

  • Anonymous
    November 06, 2013
    What is the Microsoft.WindowsAzure.Storage.DataMovement assembly that ships with AzCopy?  There doesn't seem to be any information out there on it, other than it exists.

  • Anonymous
    November 11, 2013
    @Reggie This Data Movement assembly is an internal library used by AzCopy today. We are considering to expose this library so that customer can introduce more data movement capabilities in their own applications, however we don’t have a date to share yet.

  • Anonymous
    March 26, 2014
    Any word yet on sharing the code or assembly for this? Honestly, you are getting throughputs that are 3X-5X faster than we get with our code. Thanks, Mark

  • Anonymous
    March 27, 2014
    Hi Mark, We will consider to open source AzCopy in the future, but do not have a timeline yet. Regards Zhiming

  • Anonymous
    April 22, 2014
    Is there a way to know how many threads AzCopy is executing.  I started to copy 2 small files (3GB each) from a directory as a test and I'm trying to get some statistics.  I'm using /V switch but I don't believe it will provide the statistics I need.

  • How many threads is AzCopy executing
  • Time taken to copy the files

  • Anonymous
    April 22, 2014
    Hi Gus, Thanks for your feedback. AzCopy will by default start CORE*8 concurrent operations to copy the blobs, you can use option /nc to change the number. And AzCopy does not show ‘time taken to copy the files’, we will add it to the list of feature request. Zhiming

  • Anonymous
    June 12, 2014
    Is it possible to use this tool to copy container content between two Storage Emulators on two machines (in the same LAN)?

  • Anonymous
    June 13, 2014
    Hello Gabriel Access to emulator outside of localhost is not a supported scenario.   Samarth

  • Anonymous
    June 15, 2014
    Hi there, if I was to transfer some VHDs from one subscription to another WITHIN the same datacentre, what would the data charges be? Thanks

  • Anonymous
    June 15, 2014
    Can I copy files from storage emulator to an azure storage account?  Having problems connectting to storage emulator account.

  • Anonymous
    June 16, 2014
    Hi J, There are no Bandwidth charges for ingress and egress within the same region, so the only charges would be transaction charges and the capacity used, as described here: blogs.msdn.com/.../understanding-windows-azure-storage-billing-bandwidth-transactions-and-capacity.aspx. Zhiming

  • Anonymous
    June 18, 2014
    Hi Zach, It is not possible to transfer from storage emulator to cloud directly, but you could download all the files to local disk and then upload them to the cloud storage account. Zhiming

  • Anonymous
    July 17, 2014
    can we initiate the download from SSMS or SQL agent job

  • Anonymous
    July 19, 2014
    The comment has been removed

  • Anonymous
    July 23, 2014
    hi, i have many containers(100+ ) in a storage account and each of those containers have many blobs in them. I want to copy all these containers and the blobs to a different storage account.   AzCopy https://<sourceaccount>.blob.core.windows.net/ https://<destaccount>.blob.core.windows.net/  /sourcekey:<key> /destkey:<key> /S will the above command copy all the containers and the blobs?

  • Anonymous
    July 24, 2014
    Hi JAW, AzCopy does not support to copy whole storage account's data yet, you must explicitly specify the container in the 'source'. We will put this request into our feature list. Zhiming

  • Anonymous
    July 31, 2014
    Hi Team, My customer has below query:- We are moving data from Azure VM to Azure storage account using AzCopy. Both VM and Storage account is in Same region. we have got ~60 MBps speed for 1GB of 50 files, and ~1Mbps speed for 200MB of 10000 files. What are the commands for AzCopy to parallelize this to allow more throughput? Any response would be appreciate.

  • Anonymous
    August 01, 2014
    Hi Deva, AzCopy by default starts CORE * 8 concurrent operations to transfer data in parallel, you can specify option /nc:<number of concurrent operations> to change the default value, and the throughput is also limited by the size of the VM you are using. Zhiming

  • Anonymous
    September 07, 2014
    Hi, can I move a VHD from one container to another, either within the same storage account or a different storage account? When I capture a VM, I don't believe I can specify the container so end up with many in the same container. I then want to copy a VHD to a colleague but all will be copied. Is there a way around this?

  • Anonymous
    September 08, 2014
    The comment has been removed

  • Anonymous
    October 12, 2014
    Hello, Is it possible to run this within Task sceduler. I can't seem to be able to open up AzCopy in a task.

  • Anonymous
    October 13, 2014
    The comment has been removed

  • Anonymous
    October 13, 2014
    The comment has been removed

  • Anonymous
    October 14, 2014
    The comment has been removed

  • Anonymous
    November 03, 2014
    It would be great if you could include all Azcopy parms on a single page somewhere and make it easily findable

  • Anonymous
    November 03, 2014
    Hi, Thanks for your feedback, we have one single page to introduce all the details about AzCopy azure.microsoft.com/.../storage-use-azcopy (short link: http://aka.ms/azcopy). Zhiming

  • Anonymous
    December 09, 2014
    Hello, Can i use AzCopy to copy/move whole storage account without mention container names. Thanks

  • Anonymous
    December 09, 2014
    Hi Jalal, AzCopy does not support for copying whole storage account yet, we have put this requirement into our feature list, but no concrete timeline to share yet. You may want to use Azure Powershell to iterate all the containers within a storage account and call AzCopy to copy each container. Zhiming