Speeding up data copy service from TrueNAS to Azure Databox

Randy Burgess 0 Reputation points
2023-10-09T18:56:21.2933333+00:00

We are transferring around 300 TB of data from a TrueNAS storage appliance to multiple Azure DataBox devices.

We're using the Copy data (Preview) feature to transfer the data, but it's fairly slow (from our perspective). We're getting maybe 2 TB of transfer over 24 hr per data copy service job that's running.

We have four databoxes running 2-3 data copy service jobs all at the same time and none of the jobs seem to negatively impact the speed of each other. When we ran the first job overnight, by itself, the speed was the same.

Questions

  1. Are there any tips to speed the transfer of a data copy service up?
  2. Why do multiple data copy services run at the same speed?
  3. Is the transfer software installed on the Databoxes the bottleneck?
Azure Data Box
Azure Data Box
A family of appliances and solutions for offline data transfer to Azure​.
39 questions
Azure Storage Accounts
Azure Storage Accounts
Globally unique resources that provide access to data management services and serve as the parent namespace for the services.
2,872 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sumarigo-MSFT 44,906 Reputation points Microsoft Employee
    2023-11-06T04:00:30.7166667+00:00

    @@Randy Burgess Thanks for providing the required information:

    For (1), I was also trying to understand what type of physical media (like HDD or SSD etc). So, RAID 5+0 with how many disks? 

    For (4), I don’t think it’s a good idea for same databox to have multiple jobs do the same hierarchy copy. What I mean is:

    Job 1: Copy from \\truenas\dataset to \\databox\blockblob_archive\dataset

    Job 2: Copy from \\truenas\dataset to \\databox\blockblob_archive\dataset

    Etc

    Instead, better one would be to do a separation:

    Job 1: Copy from \\truenas\dataset\datadir1 to \\databox\blockblob_archive\dataset1

    Job 1: Copy from \\truenas\dataset\datadir2 to \\databox\blockblob_archive\dataset2

     For (5), is 220MBps sustained over time from the storage for reads? i.e. for a larger copy.

     
    Anyways, if this issue is still persistent, can you please contact support, so If you have a support plan, I request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support. In this case, could you send an email to AzCommunity[at]Microsoft[dot]com referencing this thread as well as your subscription ID. Please mention "ATTN subm" in the subject field. Thank you for your cooperation on this matter and look forward to your reply. and work on getting the support package from the device? Given that, it has been few weeks, are you finished copying the data? Or is this a PoC that they are trying to do?

    Please let us know if you have any further queries. I’m happy to assist you further.

    Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.