TIP

💡 Learn more: AzCopy (opens new window)

This post was brought to you by Kumar Allamraju (opens new window).

Move your data from AWS S3 to Azure Blob Storage using AzCopy

Introduction

Customers who wanted to migrate their data from AWS S3 to Azure Blob/File Storage have encountered many challenges. They had to come with their own tools, or possibly write a script between the cloud providers to read the data from AWS and move it in Azure Blob/File Storage. This meant the scale and speed of the data transfer was limited to the client in the middle adding to the complexity of the move. AzCopy comes to the rescue.

Azure has released an AzCopy command line tool (MacOS, Linux and Windows) to simplify the transfer process between AWS S3 to Azure Blob storage. It is the next generation data transfer utility for Azure Blob/File Storage, has been redesigned from scratch to provide data movement at greater scale with built-in resiliency. AzCopy supports copying data efficiently both from a local file system to Azure Blob Storage and between Azure Storage accounts. The latest release added support for AWS S3 as a source to help you move your data using a simple command.

Let's Get Started

1. Download Azcopy (opens new window)

Open a command prompt, type the following command, and then press the ENTER key.

2. azcopy login

3. Use the following syntax to copy an AWS S3 bucket to a Azure Blob container:

azcopy cp "https://s3.amazonaws.com/mybucket/" "https://mystorageaccount.blob.core.windows.net/mycontainer<SAS>" --recursive

Please make sure the AWS S3 bucket and the Blob/File Storage container is publicly accessible. Here are the steps (opens new window) to generate SAS for your storage container.

Conclusion

In our internal testing, we have seen transfer rates of 50 GB and higher when the AWS S3 bucket and Azure Storage account are in the same region. This level of performance makes AzCopy a fast and simple option when you want to move large amounts of data from AWS. In addition to the higher speeds, AzCopy also provides resiliency. Each failure is automatically retried a number of times to mitigate network glitches. A failed or canceled job can be resumed or restarted so that you can easily move TBs of data at once. Go ahead and give it a try and share your feedback with us.

Feel free to check out earlier tips on AzCopy at AzureTipsAndTricks (opens new window)

Working with AzCopy (opens new window)

Prevent AzCopy Uploads from maxing out Internet Connection Speed (opens new window)