What's the fastest way to copy large folders from one server to another?

Broadway, Joel 20 Reputation points
2023-09-06T12:17:08.9733333+00:00

I am using robocopy to copy data from an old server to a new server. We have three very large folders that get thousands of new and modified files every day. What is the fastest way to robocopy the new and modified files from the three folders?

  1. Open three command windows and robocopy each folder simultaneously?
  2. Robocopy Folder 1, then Folder 2, etc. in one command window using multi-thread options? What are the guidelines for using setting -mt option based on processors, cores, etc.?
Windows Server
Windows Server
A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.
12,518 questions
Windows Server Migration
Windows Server Migration
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Migration: The process of making existing applications and data work on a different computer or operating system.
413 questions
0 comments
Accepted answer
  1. Muhammad Binyameen 395 Reputation points
    2023-09-06T16:40:55.05+00:00

    To achieve the fastest and most efficient copying of new and modified files using Robocopy, I would recommend using multiple instances of Robocopy simultaneously in separate command windows, one for each folder. This approach will utilize the available resources better and take advantage of parallel processing to speed up the overall copying process.

    However, before you start copying, make sure that the destination folder on the new server is empty or doesn't contain any conflicting files. This avoids any potential issues or conflicts during the copying process.

    Regarding the '-MT' option in Robocopy, it is used to enable multi-threaded copying, which can significantly improve performance. The value you should use for the '-MT' option depends on the number of available processors or cores on your server. The general guideline is to use a value equal to or slightly less than the number of processors/cores.

    For example, if your server has 8 processors/cores, you can use '-MT:8' or '-MT:4' to specify either 8 or 4 concurrent threads. It's recommended to experiment with different values and observe the performance to determine the best setting for your specific server configuration. Also, keep in mind that using too many threads may cause excessive disk I/O, which could impact overall performance.

    In summary, to achieve the fastest copying of new and modified files using Robocopy, open multiple command windows and run Robocopy for each folder simultaneously. Use the '-MT' option with a value equal to or slightly less than the number of available processors/cores on your server. Remember to ensure the destination folder is empty or doesn't contain conflicting files before starting the copying process.

    2 people found this answer helpful.
    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Matt-Radcliffe 5 Reputation points
    2023-11-30T11:06:50.76+00:00

    It depends on your specific hardware and network configuration. However, in general, using multiple threads can improve performance, especially for large folders with a large number of new and modified files.

    Opening three command windows and robocopying each folder simultaneously is a viable option, especially if you have enough spare CPU and network resources to handle the load. This approach will allow each folder to be copied independently, which can maximize throughput.

    Robocopying Folder 1, then Folder 2, etc. in one command window using multi-thread options can also be effective, but it may not be as fast as opening multiple command windows. The optimal number of threads will depend on your hardware configuration. As a general rule of thumb, you should use no more than half of your available CPU cores, as using too many threads can actually decrease performance.

    The best way to determine the fastest way to robocopy your data is to experiment with different options and see what works best for your specific environment.

    0 comments