AZCopy and S3 Authentication

Zachary Legault 1 Reputation point
2021-08-19T03:48:33.62+00:00

We're trying to use AZCopy or MSE to transfer data from an S3 bucket into a storage blob but we're having issues getting AZCopy to authenticate with the S3 bucket. We get the following error:

failed to perform copy command due to error: s3 authentication to <s3 url>.s3-ap-southeast-2.amazonaws.com is not currently suported in AzCopy

We can't seem to find any resource that says exactly where S3 authentication is supported. Is there any info on how we can work around this/find out when this may be supported?

Azure Storage Explorer
Azure Storage Explorer
An Azure tool that is used to manage cloud storage resources on Windows, macOS, and Linux.
240 questions
Azure Blob Storage
Azure Blob Storage
An Azure service that stores unstructured data in the cloud as blobs.
2,578 questions
0 comments

2 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Rinshad R 1 Reputation point
    2021-08-19T04:54:33.33+00:00

    Hi Zachary,

    Could you please confirm, as mentioned in the MSFT document https://video2.skills-academy.com/en-us/azure/storage/common/storage-use-azcopy-s3, did you set the environment variable? This environment variable helps you to authenticate with AWS S3.

    Thanks
    Rinshad.R

  2. Sumarigo-MSFT 44,996 Reputation points Microsoft Employee
    2021-08-19T12:46:05.077+00:00

    @Zachary Legault Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

    Adding more information to the above response! AzCopy uses an access key, and secret to authenticate with AWS S3. For the destination Blob storage account you can use any of the available authentication options (SAS token, or Azure Active Directory authentication). See here for more information on azcopy aws azure

    To copy an S3 bucket to a Blob container, use the following command: 

    azcopy cp "https://s3.amazonaws.com/mybucket/" "https://mystorageaccount.blob.core.windows.net/mycontainer<SAS>" --recursive

    AzCopy V10 uses the service APIs to read the source data directly into Storage, i.e. the Storage must be able to read the S3 data directly.

    Alternatively, if you are looking for a fully managed Platform-as-a-Service (PaaS) option for migrating data from AWS S3 to Azure Storage, consider Azure Data Factory (ADF), which provides more additional benefits: [see here][2]

    If the issue still persist, share the commands and screenshot of the error message.

    Looking forward for your reply!

    Kindly let us know if the above helps or you need further assistance on this issue.

    ------------------------------------------------------------------------------------------------------------------------------------------------------------------

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

    0 comments