Optimize search through all drives

123244 60

Hi, I need to implement an efficient search through user's disks. I would really like to search multiple disks in parallel to speed up the process, but at the same time, avoid using too much CPU resources.

I've already implemented a directory search that will scan a directory first superficially (only the current directory), and then, if necessary, will also scan subdirectories. Here is an example:

      public async Task SearchAsync(SearchOptions searchOptions)
      {
		  // Look in searched directory
          await ShallowSearchAsync(searchOptions);

		  // If nested search needs to be performed, look also in subdirectories
          if (searchOptions.Filter.IsNestedSearch)
          {
				await Task.Run(async () =>
	            {
	                foreach(subdirectory in EnumerateSubDirectories())
					{
						await subdirectory.SearchAsync(searchOptions);
					}
	            }, searchOptions.Token);
          }
      }

	public async Task ShallowSearchAsync(SearchOptions searchOptions)
	{
		// Here we look through directorie's files 
		// and filter out those that do not meet the search conditions
	}

If you have any ideas on how to optimize my search method, I'd be glad to hear them.

Junjie Zhu - MSFT 18,976 Reputation points Microsoft Vendor

2024-08-27T03:09:30.93+00:00

Hi ,

Welcome to Microsoft Q&A!

Is your project a UWP project? UWP runs in a sandbox and cannot access external files by default.

Thank you.
123244 60 Reputation points

2024-08-27T07:13:37.1+00:00

Hi, @Junjie Zhu - MSFT , I'm writing a Win UI project, using System.IO to access the file system.

Thank you for responding.
Junjie Zhu - MSFT 18,976 Reputation points Microsoft Vendor

2024-08-27T09:27:27.65+00:00

It is difficult to improve much using system.IO.
Maybe you could try to use multiple Tasks or recursive methods.
123244 60 Reputation points

2024-08-27T10:03:43.0366667+00:00

I'm sorry for the inaccuracy in my question. I'm looking for a way to manage threads. It is necessary that all disks (threads where they are searched) have a certain limit of operations and the fewer disks are searched, the more operations are allowed. This means that at the beginning the operations will be a little slower, but in parallel, and at the end they will be faster, because there are fewer threads to share resources with.

I had an idea to use a semaphore, but it puts a limit on the number of threads, which is not exactly what I would like. A task manager that switches between disks at a certain time would be better for this case. If you know of something similar, I would be happy to hear about it.

Still, thanks for helping me with this.
123244 60 Reputation points

2024-08-27T10:05:33.62+00:00

I'm sorry for the inaccuracy in my question. I'm looking for a way to manage threads. It is necessary that all disks (threads where they are searched) have a certain limit of operations and the fewer disks are searched, the more operations are allowed. This means that at the beginning the operations will be a little slower, but in parallel, and at the end they will be faster, because there are fewer threads to share resources with.

I had an idea to use a semaphore, but it puts a limit on the number of threads, which is not exactly what I would like. A task manager that switches between disks at a certain time would be better for this case. If you know of something similar, I would be happy to hear about it.

Still, thanks for helping me with this.
Castorix31 86,036 Reputation points

2024-08-27T12:34:42.17+00:00

The fastest method to search files is to read the MFT (FSCTL_QUERY_USN_JOURNAL and so on)
123244 60 Reputation points

2024-09-02T11:27:03.85+00:00

Thank you very much, I will take your response into consideration

Share via

Optimize search through all drives

Your answer