Long Paths in .NET, Part 2 of 3: Long Path Workarounds [Kim Hamilton]

  • [アーティクル]

For now, our suggested workaround for users that encounter the MAX_PATH issue is to rearrange directories so that the names are shorter. This may sound like a cop out, but this is ultimately easier on users because of (1) limited tool support (i.e. Explorer doesn't work with long paths) and (2) getting the full System.IO functionality for long paths results in a significant code delta for users. However, if you really want to work with paths longer than MAX_PATH you can, and this part of the series demonstrates how.

Recall from Part 1 that if you prefix the path with \\?\ and use the Unicode versions of the Win32 APIs, you can use paths up to 32K characters in length. These code samples will use that fact to show a few common file operations with long path files.

Deleting a File

Let's start with the simplest example – deleting a file. Recall that Explorer won't let you delete long path files, so you'll need this to clean up the files you create in the subsequent section.

First, we look at the Win32 API docs for DeleteFile and confirm that it supports long paths. DeleteFile does according to this comment:

In the ANSI version of this function, the name is limited to MAX_PATH characters. To extend this limit to 32,767 wide characters, call the Unicode version of the function and prepend "\\?\" to the path. For more information, see Naming a File.

So we specify the PInvoke signature:

using System;

using System.Runtime.InteropServices;

 

[DllImport("kernel32.dll", CharSet = CharSet.Unicode)]

[return: MarshalAs(UnmanagedType.Bool)]

internal static extern bool DeleteFile(string lpFileName);

And then all we have to do is call it with a file name prefixed by \\?\:

// This code snippet is provided under the Microsoft Permissive License.

public static void Delete(string fileName) {

    string formattedName = @"\\?\" + fileName;

    DeleteFile(formattedName);

}

For some tasks such as deleting, moving, and renaming a file, you simply PInvoke to the Win32 APIs and you're done. For other cases, such as writing to a file, you mix the Win32 calls with the System.IO APIs.

Writing to or Reading from a file

First you need to create or open a file with the Win32 CreateFile function. CreateFile returns a file handle, which you can pass to a System.IO.FileStream constructor. Then you simply work with the FileStream as normal.

// This code snippet is provided under the Microsoft Permissive License.

using System;

using System.IO;

using System.Runtime.InteropServices;

using Microsoft.Win32.SafeHandles;

 

[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]

internal static extern SafeFileHandle CreateFile(

    string lpFileName,

    EFileAccess dwDesiredAccess,

    EFileShare dwShareMode,

    IntPtr lpSecurityAttributes,

    ECreationDisposition dwCreationDisposition,

    EFileAttributes dwFlagsAndAttributes,

    IntPtr hTemplateFile);

 

public static void TestCreateAndWrite(string fileName) {

 

    string formattedName = @"\\?\" + fileName;

    // Create a file with generic write access

    SafeFileHandle fileHandle = CreateFile(formattedName,

        EFileAccess.GenericWrite, EFileShare.None, IntPtr.Zero,

        ECreationDisposition.CreateAlways, 0, IntPtr.Zero);

 

    // Check for errors

    int lastWin32Error = Marshal.GetLastWin32Error();

    if (fileHandle.IsInvalid) {

        throw new System.ComponentModel.Win32Exception(lastWin32Error);

    }

 

    // Pass the file handle to FileStream. FileStream will close the

    // handle

    using (FileStream fs = new FileStream(fileHandle,

                                    FileAccess.Write)) {

        fs.WriteByte(80);

        fs.WriteByte(81);

        fs.WriteByte(83);

        fs.WriteByte(84);

    }

}

This sample shows writing a few bytes, but once you have the FileStream, you can do anything you would normally do: wrap it in a BinaryWriter, etc.

If you wanted to open a file instead of creating it, you would change the creation disposition from CreateAlways to OpenExisting. If you wanted to read a file instead of writing, you would change the file access from GenericWrite to GenericRead.

See the end of the article for definitions of the enums and structs in this example.

Finding Files and Directories

So far the workarounds have been fairly minor, but suppose you want to get the files and folders contained in a folder. Unfortunately, now you're starting to rewrite the .NET libraries.

// This code snippet is provided under the Microsoft Permissive License.

using System;

using System.Collections.Generic;

using System.IO;

using System.Runtime.InteropServices;

using Microsoft.Win32.SafeHandles;

 

[DllImport("kernel32.dll", CharSet = CharSet.Unicode)]

internal static extern IntPtr FindFirstFile(string lpFileName, out

                                WIN32_FIND_DATA lpFindFileData);

 

[DllImport("kernel32.dll", CharSet = CharSet.Unicode)]

internal static extern bool FindNextFile(IntPtr hFindFile, out

                                WIN32_FIND_DATA lpFindFileData);

 

[DllImport("kernel32.dll", SetLastError = true)]

[return: MarshalAs(UnmanagedType.Bool)]

internal static extern bool FindClose(IntPtr hFindFile);

 

// Assume dirName passed in is already prefixed with \\?\

public static List<string> FindFilesAndDirs(string dirName) {

 

    List<string> results = new List<string>();

    WIN32_FIND_DATA findData;

    IntPtr findHandle = FindFirstFile(dirName + @"\*", out findData);

 

    if (findHandle != INVALID_HANDLE_VALUE) {

        bool found;

        do {

            string currentFileName = findData.cFileName;

 

            // if this is a directory, find its contents

            if (((int)findData.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) != 0) {

                if (currentFileName != "." && currentFileName != "..")

                {

                List<string> childResults = FindFilesAndDirs(Path.Combine(dirName, currentFileName));

                // add children and self to results

                results.AddRange(childResults);

                results.Add(Path.Combine(dirName, currentFileName));

                }

            }

 

            // it's a file; add it to the results

            else {

                results.Add(Path.Combine(dirName, currentFileName));

            }

 

            // find next

            found = FindNextFile(findHandle, out findData);

        }

        while (found);

    }

 

    // close the find handle

    FindClose(findHandle);

    return results;

}

Constants, Structs and Enums for the code samples

internal static IntPtr INVALID_HANDLE_VALUE = new IntPtr(-1);

internal static int FILE_ATTRIBUTE_DIRECTORY = 0x00000010;

internal const int MAX_PATH = 260;

 

[StructLayout(LayoutKind.Sequential)]

internal struct FILETIME {

    internal uint dwLowDateTime;

    internal uint dwHighDateTime;

};

 

[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]

internal struct WIN32_FIND_DATA {

    internal FileAttributes dwFileAttributes;

    internal FILETIME ftCreationTime;

    internal FILETIME ftLastAccessTime;

    internal FILETIME ftLastWriteTime;

    internal int nFileSizeHigh;

    internal int nFileSizeLow;

    internal int dwReserved0;

    internal int dwReserved1;

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = MAX_PATH)]

    internal string cFileName;

    // not using this

    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 14)]

    internal string cAlternate;

}

 

[Flags]

public enum EFileAccess : uint {

    GenericRead = 0x80000000,

    GenericWrite = 0x40000000,

    GenericExecute = 0x20000000,

    GenericAll = 0x10000000,

}

 

[Flags]

public enum EFileShare : uint {

    None = 0x00000000,

    Read = 0x00000001,

    Write = 0x00000002,

    Delete = 0x00000004,

}

 

public enum ECreationDisposition : uint {

    New = 1,

    CreateAlways = 2,

    OpenExisting = 3,

    OpenAlways = 4,

    TruncateExisting = 5,

}

 

[Flags]

public enum EFileAttributes : uint {

    Readonly = 0x00000001,

    Hidden = 0x00000002,

    System = 0x00000004,

    Directory = 0x00000010,

    Archive = 0x00000020,

    Device = 0x00000040,

    Normal = 0x00000080,

    Temporary = 0x00000100,

    SparseFile = 0x00000200,

    ReparsePoint = 0x00000400,

    Compressed = 0x00000800,

    Offline = 0x00001000,

    NotContentIndexed = 0x00002000,

    Encrypted = 0x00004000,

    Write_Through = 0x80000000,

    Overlapped = 0x40000000,

    NoBuffering = 0x20000000,

    RandomAccess = 0x10000000,

    SequentialScan = 0x08000000,

    DeleteOnClose = 0x04000000,

    BackupSemantics = 0x02000000,

    PosixSemantics = 0x01000000,

    OpenReparsePoint = 0x00200000,

    OpenNoRecall = 0x00100000,

    FirstPipeInstance = 0x00080000

}

 

[StructLayout(LayoutKind.Sequential)]

public struct SECURITY_ATTRIBUTES {

    public int nLength;

    public IntPtr lpSecurityDescriptor;

    public int bInheritHandle;

}

Update: The SizeConst of the WIN32_FIND_DATA.cAlternate member incorrectly stated 10 instead of 14 and has been revised.

Comments

  • Anonymous
    March 26, 2007
    PingBack from http://blogs.msdn.com/bclteam/archive/2007/03/26/long-paths-in-net-part-2-of-3-long-path-workarounds-kim-hamilton.aspx

  • Anonymous
    March 26, 2007
    The BCL team has been blogging recently about long file path support in the .NET Framework (or lack thereof).

  • Anonymous
    March 26, 2007
    Thank you for part 2 of this series (it's been a while since part 1 ...), especially for all the usefull P/Invoke snippets. I'm wondering if it's time Microsoft came up with Microsoft.Win32.Kernel.dll and similar framework assemblies. Since everybody is doing it, why not give them something that does it right? But the real question: Did you leave out CAS because it doesn't support long file names?

  • Anonymous
    March 26, 2007
    Hi Henry, Responses inline... >>Thank you for part 2 of this series (it's been a while since part 1 ...), So sorry for the delay, it's been a busy month. But lots of fun and exciting stuff which we'll no doubt be blogging about soon. :) >> I'm wondering if it's time Microsoft came up with Microsoft.Win32.Kernel.dll and similar framework assemblies. Since everybody is doing it, why not give them something that does it right? One problem is that the demand for long paths has not been strong enough yet. Because of compatibility problems described in part 1, this will be a significant amount of effort on the win32 side (mostly on the migration/mitigation front). But, as you pointed out, an obvious problem is the longer we put it off, the more individual workarounds may happen. And it's lots of additional work for users to get the workarounds right. For example, in my samples I left out file name canonicalization, which would be necessary for production code. You're definitely right that Microsoft needs to address this. The big issue is when does it become pressing enough that there's the go-ahead to address this on a larger scale. >> But the real question: Did you leave out CAS because it doesn't support long file names? Actually, CAS is fine with long file names, so you could definitely add it on top. I left CAS out of these samples because I wanted to show only the essential steps to save space. Another important thing I left out: I'm not canonicalizing the file name via win32 GetFullPathName. Since ? turns off win32 file name canonicalization, this code sample will let you create files with noncanonical names, e.g. with spaces at the end. This is a whole other topic which should probably be part 4 of the series. I realize these samples are sparse; feel free to request related APIs and I'll post those too. Thanks, Kim

  • Anonymous
    March 27, 2007
    So let me get this straight: you're putting off long file name support because there is not enough strong demand. A lot of demand, but just not strong enough. Does that mean that instead of requesting and asking for features, developers need to strongly demand them from MS? ;-)

  • Anonymous
    March 27, 2007
    There definitely are efforts to address this within MS, touched on in part 1. And you'll see more along these lines in Justin's part 3 post. The question is the larger scale long path support, which presents huge compatibility challenges (briefly discussed in the first post but I'm thinking I should elaborate on this even more in a subsequent post). And for that, it's not that riots have to occur or anything...although it would be cool if people got that excited about MS features. :) It's just the trade off. There are tons of cool features we'd love to do -- so many that we have to prioritize.

  • Anonymous
    April 12, 2007
    I'm having trouble getting your code to work. I'm trying to open the following file: "?Y:LCCP - Language and LiteraturePQ0001-9999 - French literature - Italian literature - Spanish literature - Portuguese literaturePQ0001-3999 - French literaturePQ1300-1595 - Old French literaturePQ1411-1545 - To 1350.1400PQ1425.A3 E5 1993 - Amadas and Ydoine.pdf" It keeps coming up with this error message: "The system cannot find the file specified". Any ideas?

  • Anonymous
    April 15, 2007
    Hi Jason, Are you trying OpenExisting for an existing file, where fileName is the name you specified, in a call like this? SafeFileHandle fileHandle = LongPathLibrary.CreateFile(fileName, EFileAccess.GenericAll, EFileShare.None, ECreationDisposition.OpenAlways); Can you send the code snippet you're trying? Also, I imagine Y: is a mapped network drive, right? Please send it to bclpub@microsoft.com so we can iterate offline and I'll post a follow-up comment here if it's something other readers would be interested in. Thanks, Kim

  • Anonymous
    April 18, 2007
    Just for the record, the officially suggested work-around or path renaming is not an option in many circumstances. One glaring example is in the Electronic Discovery and Computer Forensics arena.  Obviously if you are collecting files in response to a preservation order, discovery request, or subpoena, you can't go mucking about with directory names or file names.  The file system meta-data of discoverable files usually must be maintained and preserved.  This includes the original path name, file name, and usually the three time stamps (MAC times) of the files. So a working solution for the long pathname problem is required.  Obviously it would be preferable to have a solution "built-in" to the System.IO namespace.  Short of that we are currently forced to limp along with a bunch of P/Invokes that can be difficult to implement cleanly and safely. What would be very helpful in the interim would be if you guys could put together a wrapper class that would incorporate all of the Win32 IO calls that are able to take the ? path names (i.e., FindFirstFileW, FindNextFileW, GetFileAttributesW, CopyFileEx, CopyFileW, CreateFileW, GetFileSizeEx, DeleteFileW, CreateDirectoryW, GetFileTime, SetFileTime, etc...). I know this is asking a lot, but having class library created by people who are seriously "in the know" would be very beneficial to developers who have been wrangling with this issue for quite some time now. -Steve (No, I am not THE Steve Gibson.  I am THE OTHER Steve Gibson.)

  • Anonymous
    April 18, 2007
    Hi Steve, Thank you, that's excellent feedback. One issue we've been researching is who's affected by the MAX_PATH limitation and how. Computer forensics is a great example of a problem for which you can't workaround by rearranging. We've been looking into supporting long paths in the next release of the framework, but that's a couple years away. So for the meantime, we've actually been investigating ways to do exactly what you suggest -- release at least some wrapper library through codeplex, or some similar fashion. We'll definitely keep everyone posted on those efforts. And it's great to know there's a need for this. Thanks, Kim

  • Anonymous
    April 24, 2007
    Kim, May I translate this topic and post its Chinese version on my blog? Thanks, TC

  • Anonymous
    April 25, 2007
    Hi TC, Yes, that would be great. Thanks, Kim

  • Anonymous
    June 11, 2008
    The source of the main title is an inside joke I am probably not going to ever explain within the blog.

  • Anonymous
    July 07, 2008
    Updated 6/10/08 2:20pm: clarified details of proposed solution Here it is, Part 3 of the long path series

  • Anonymous
    February 20, 2009
    Finding Path lengths greater than MAX_PATH

  • Anonymous
    February 20, 2009
    Finding Path lengths greater than MAX_PATH

  • Anonymous
    February 21, 2009
    Finding Path lengths greater than MAX_PATH

  • Anonymous
    February 21, 2009
    Finding Path lengths greater than MAX_PATH