CancelIoEx spuriously fails with ERROR_NOT_FOUND when there is actually pending IO

Ilia - 1 Reputation point
2020-10-04T20:19:54.547+00:00

Hi! This ^^ leads to "access violation executing location" exception on TpIo thread when the pending IO operation really completes:

    00007ff7c000014b()  Unknown No symbols loaded.  
>   kernel32.dll!BasepTpIoCallback()    Unknown Symbols loaded.  

    ntdll.dll!TppIopExecuteCallback()   Unknown Symbols loaded.  
    ntdll.dll!TppWorkerThread() Unknown Symbols loaded.  
    kernel32.dll!BaseThreadInitThunk()  Unknown Symbols loaded.  
    ntdll.dll!RtlUserThreadStart()  Unknown Symbols loaded.

The crash happens in the following code, when it calls _guard_dispatch_icall_nop and then jumps to the address taken from the freed OVERLAPPED*:

BasepTpIoCallback:  
00007FFB30B1BAC0 48 89 5C 24 08       mov         qword ptr [rsp+8],rbx    
00007FFB30B1BAC5 48 89 6C 24 10       mov         qword ptr [rsp+10h],rbp    
00007FFB30B1BACA 48 89 74 24 18       mov         qword ptr [rsp+18h],rsi    
00007FFB30B1BACF 57                   push        rdi    
00007FFB30B1BAD0 48 83 EC 40          sub         rsp,40h    
00007FFB30B1BAD4 33 C0                xor         eax,eax    
00007FFB30B1BAD6 49 8B D9             mov         rbx,r9    
00007FFB30B1BAD9 49 8B F0             mov         rsi,r8    
00007FFB30B1BADC 48 8B FA             mov         rdi,rdx    
00007FFB30B1BADF 48 8B E9             mov         rbp,rcx    
00007FFB30B1BAE2 41 39 01             cmp         dword ptr [r9],eax    
00007FFB30B1BAE5 0F 8C CD 82 01 00    jl          CreateToolhelp32Snapshot+0D9C8h (07FFB30B33DB8h)    
00007FFB30B1BAEB 48 8B 4C 24 70       mov         rcx,qword ptr [rsp+70h]    
00007FFB30B1BAF0 44 8B C8             mov         r9d,eax    
00007FFB30B1BAF3 48 8B 57 18          mov         rdx,qword ptr [rdi+18h]    
00007FFB30B1BAF7 4C 8B C6             mov         r8,rsi    
00007FFB30B1BAFA 48 8B 07             mov         rax,qword ptr [rdi]    
00007FFB30B1BAFD 48 89 4C 24 28       mov         qword ptr [rsp+28h],rcx    
00007FFB30B1BB02 48 8B 4B 08          mov         rcx,qword ptr [rbx+8]    
00007FFB30B1BB06 48 89 4C 24 20       mov         qword ptr [rsp+20h],rcx    
00007FFB30B1BB0B 48 8B CD             mov         rcx,rbp    
00007FFB30B1BB0E FF 15 04 77 06 00    call        qword ptr [__guard_dispatch_icall_fptr (07FFB30B83218h)]    
-> 00007FFB30B1BB14 48 8B 5C 24 50       mov         rbx,qword ptr [rsp+50h]    
00007FFB30B1BB19 48 8B 6C 24 58       mov         rbp,qword ptr [rsp+58h]    
00007FFB30B1BB1E 48 8B 74 24 60       mov         rsi,qword ptr [rsp+60h]    
00007FFB30B1BB23 48 83 C4 40          add         rsp,40h    
00007FFB30B1BB27 5F                   pop         rdi

AFAIK there are no docs or examples that prohibit freeing it after ERROR_NOT_FOUND. https://video2.skills-academy.com/en-us/windows/win32/fileio/cancelioex-func just says if the function succeeds I "*must not free or reuse the OVERLAPPED structure associated with the canceled I/O operations until they have completed*", but there is no such requirement in case of ERROR_NOT_FOUND.

The situation is similar in other places, but they also say nothing about ERROR_NOT_FOUND: this page https://video2.skills-academy.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o says "*do not deallocate or modify the OVERLAPPED structure or the data buffer until all asynchronous I/O operations to the file object have been completed*", and this page https://video2.skills-academy.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations says "*once the I/O operation completes (either successfully or with a canceled status) then the overlapped structure is no longer in use by the system and can be reused*".

In addition, Canceling Asynchronous I/O example (https://video2.skills-academy.com/en-us/windows/win32/fileio/canceling-pending-i-o-operations) doesn't wait operation to complete when CancelIoEx() returns ERROR_NOT_FOUND:

result = CancelIoEx( hFile, lpOverlapped );  

*pbCancelCalled = TRUE;  

if (result == TRUE || GetLastError() != ERROR_NOT_FOUND)   
{  
    // Wait for the I/O subsystem to acknowledge our cancellation.  
    // Depending on the timing of the calls, the I/O might complete with a  
    // cancellation status, or it might complete normally (if the ReadFile was  
    // in the process of completing at the time CancelIoEx was called, or if  
    // the device does not support cancellation).  
    // This call specifies TRUE for the bWait parameter, which will block  
    // until the I/O either completes or is canceled, thus resuming execution,   
    // provided the underlying device driver and associated hardware are functioning   
    // properly. If there is a problem with the driver it is better to stop   
    // responding here than to try to continue while masking the problem.  

    result = GetOverlappedResult( hFile, lpOverlapped, lpNumberOfBytesRead, TRUE );  

    // ToDo: check result and log errors.   
}

Regarding TpIo, I use WaitForThreadpoolIoCallbacks() with fCancelPendingCallbacks=TRUE and there is a note (https://video2.skills-academy.com/en-us/windows/win32/api/threadpoolapiset/nf-threadpoolapiset-waitforthreadpooliocallbacks) that "*... only queued callbacks are canceled. Pending I/O requests are not canceled. Therefore, the caller should call GetOverlappedResult for the OVERLAPPED structure to check whether the I/O operation has completed before freeing the structure. ... Be careful not to free the OVERLAPPED structure while I/O requests are still pending; use GetOverlappedResult to determine the status of the I/O operation and wait for the operation to complete*", but in my case, firstly I do CancelIoEx() which returns ERROR_NOT_FOUND, and only then call WaitForThreadpoolIoCallbacks() function, so there should be no pending IO requests at this moment.

Thus, all of this looks like CancelIoEx() erroneously returns ERROR_NOT_FOUND when there is actually a pending IO request.

Here is my program (x64/Release, VS2017, Windows 10 Pro, version 2004, build 19041.508):

#undef NDEBUG  
#define CHECK assert  
#include <assert.h>  
#include <stdio.h>  
#include <stdlib.h>  
#include <time.h>  

#include <string>  

#include <windows.h>  

static void pipe(HANDLE& hRead, HANDLE& hWrite) {  
    GUID guid;  
    CHECK(CoCreateGuid(&guid) == S_OK);  
    std::string name = "\\\\.\\pipe\\" + std::to_string(guid.Data1) + std::to_string(guid.Data2) + std::to_string(guid.Data3) + std::to_string(*(uint64_t*)&guid.Data4[0]);  

    SECURITY_ATTRIBUTES sa = {};  
    sa.nLength = sizeof sa;  
    sa.bInheritHandle = TRUE;  

    hRead = CreateNamedPipeA(name.c_str(), PIPE_ACCESS_INBOUND | FILE_FLAG_FIRST_PIPE_INSTANCE | FILE_FLAG_OVERLAPPED, PIPE_TYPE_BYTE, 1, 4096, 4096, 0, &sa);  
    CHECK(hRead != INVALID_HANDLE_VALUE);  
    CHECK(SetHandleInformation(hRead, HANDLE_FLAG_INHERIT, 0));  

    hWrite = CreateFileA(name.c_str(), FILE_WRITE_DATA | SYNCHRONIZE, 0, &sa, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);  
    CHECK(hWrite != INVALID_HANDLE_VALUE);  
}  


void WINAPI IoCompletionCallback(PTP_CALLBACK_INSTANCE Instance, void* Context, void* Overlapped, ULONG IoResult, ULONG_PTR NumberOfBytesTransferred, PTP_IO Io)  
{  
    if (IoResult != ERROR_BROKEN_PIPE)  
    if (IoResult != ERROR_OPERATION_ABORTED)  
        printf("IoCompletionCallback err = %lu\n", IoResult);  
}  

int main(int argc, char* argv[])  
{  
    int rc;  

    if (argc >= 2 && strcmp(argv[1], "--wait-and-exit") == 0)  
    {  
        Sleep(1);  
        return 0;  
    }  

    srand(time(NULL));  

    for (int i = 0; i < 100000; ++i)  
    {  
        HANDLE hPipeRead, hChildStdOut;  
        pipe(hPipeRead, hChildStdOut);  

        // start read aio  

#define BUFFER_SIZE 1048576  
        char* buffer = new char[BUFFER_SIZE];  
        OVERLAPPED* overlapped = new OVERLAPPED{};  
        printf("%d: ovlp %p\n", i, overlapped);  

        PTP_IO tpio = CreateThreadpoolIo(hPipeRead, IoCompletionCallback, NULL, NULL);  
        CHECK(tpio != NULL);  
        StartThreadpoolIo(tpio);  

        rc = ReadFile(hPipeRead, buffer, BUFFER_SIZE, NULL, overlapped);  
        CHECK(!rc && GetLastError() == ERROR_IO_PENDING);  

        // start process and wait for it to exit  

        HANDLE hChildProcess;  
        {  
            std::string cmdl = std::string(argv[0]) + " --wait-and-exit";  
            STARTUPINFO si = {};  
            si.cb = sizeof si;  
            si.dwFlags |= STARTF_USESTDHANDLES;  
            si.hStdOutput = hChildStdOut;  
            PROCESS_INFORMATION pi;  
            CHECK(CreateProcessA(NULL, (char*)cmdl.data(), NULL, NULL, TRUE, 0, NULL, NULL, &si, &pi));  
            CloseHandle(hChildStdOut);  
            CloseHandle(pi.hThread);  
            hChildProcess = pi.hProcess;  
        }  
        CHECK(WaitForSingleObject(hChildProcess, INFINITE) == WAIT_OBJECT_0);  

        // cancel read aio  

        rc = CancelIoEx(hPipeRead, overlapped);  
        if (!rc)  
        {  
            if (GetLastError() != ERROR_NOT_FOUND)  
                printf("CancelIoEx = %d, err = %lu\n", rc, GetLastError()), exit(EXIT_FAILURE);  
        }  
        else  
        {  
            DWORD dwDummy = 0;  
            rc = GetOverlappedResult(hPipeRead, overlapped, &dwDummy, TRUE);  
            if (rc || GetLastError() != ERROR_OPERATION_ABORTED)  
                printf("GetOverlappedResult = %d, err = %lu\n", rc, GetLastError()), exit(EXIT_FAILURE);  
        }  

        // verify  

        DWORD dwDummy = 0;  
        rc = GetOverlappedResult(hPipeRead, overlapped, &dwDummy, FALSE);  
        if (rc || GetLastError() != ERROR_BROKEN_PIPE)  
            printf("GetOverlappedResult = %d, err = %lu\n", rc, GetLastError()), fflush(stdout); // here!  

        // cleanup  

        CloseHandle(hChildProcess);  

        WaitForThreadpoolIoCallbacks(tpio, TRUE);  
        CloseHandle(hPipeRead);  // should be closed before CloseThreadpoolIo  
        CloseThreadpoolIo(tpio);  

        delete overlapped;  
        delete[] buffer;  
    }  

    getchar();  

    return 0;  
}

Output:

...  
880: ovlp 0000014CDA205CA0  
881: ovlp 0000014CDA205A00  
882: ovlp 0000014CDA205BE0  
883: ovlp 0000014CDA205A30  
884: ovlp 0000014CDA205F70  
885: ovlp 0000014CDA2059D0  
886: ovlp 0000014CDA205CD0  
887: ovlp 0000014CDA2059A0  
888: ovlp 0000014CDA205BB0  
889: ovlp 0000014CDA205B50  
890: ovlp 0000014CDA205F40  
891: ovlp 0000014CDA205910  
CHECK GetOverlappedResult = 0, err = 996  
892: ovlp 0000014CDA2058B0

Debugger:

Watch 1:  
        Name    Value  
        *(void**)0x07FFB30B83218    0x00007ffb30b263e0 {kernel32.dll!_guard_dispatch_icall_nop} void *  
        *(void**)0x0000014cda205910 0x00007ff7c000014b  void *  

Threads:  
Not Flagged >    0x00009028  0x00    Worker Thread   ntdll.dll!TppWorkerThread   00007ff7c000014b  
                        00007ff7c000014b()  
                        kernel32.dll!BasepTpIoCallback()  
                        ntdll.dll!TppIopExecuteCallback()  
                        ntdll.dll!TppWorkerThread()  
                        kernel32.dll!BaseThreadInitThunk()  
                        ntdll.dll!RtlUserThreadStart()  

Not Flagged     0x0000475C  0x00    Main Thread Main Thread [Inline Frame] cancelioex_test.exe!globals::destroy  
                        ntdll.dll!NtWaitForAlertByThreadId()  
                        ntdll.dll!TppBarrierAdjust()  
                        ntdll.dll!TpWaitForIoCompletion()  
                        [Inline Frame] cancelioex_test.exe!globals::destroy() Line 36  
                        cancelioex_test.exe!main(int argc, char * * argv) Line 162  
                        [Inline Frame] cancelioex_test.exe!invoke_main() Line 78  
                        cancelioex_test.exe!__scrt_common_main_seh() Line 283  
                        kernel32.dll!BaseThreadInitThunk()  
                        ntdll.dll!RtlUserThreadStart()  

Not Flagged     0x00008CE8  0x00    Worker Thread   ntdll.dll!TppWorkerThread   ntdll.dll!NtWaitForWorkViaWorkerFactory  
                        ntdll.dll!NtWaitForWorkViaWorkerFactory()  
                        ntdll.dll!TppWorkerThread()  
                        kernel32.dll!BaseThreadInitThunk()  
                        ntdll.dll!RtlUserThreadStart()  

Not Flagged     0x0000717C  0x00    Worker Thread   ntdll.dll!TppWorkerThread   ntdll.dll!NtWaitForWorkViaWorkerFactory  
                        ntdll.dll!NtWaitForWorkViaWorkerFactory()  
                        ntdll.dll!TppWorkerThread()  
                        kernel32.dll!BaseThreadInitThunk()  
                        ntdll.dll!RtlUserThreadStart()  

Not Flagged     0x000052E8  0x00    Worker Thread   ntdll.dll!TppWorkerThread   ntdll.dll!NtWaitForWorkViaWorkerFactory  
                        ntdll.dll!NtWaitForWorkViaWorkerFactory()  
                        ntdll.dll!TppWorkerThread()  
                        kernel32.dll!BaseThreadInitThunk()  
                        ntdll.dll!RtlUserThreadStart()

30005-screenshot.png

Source code: 30023-source1cpp.txt

Feedback Hub: https://aka.ms/AA9v13c

Windows API - Win32
Windows API - Win32
A core set of Windows application programming interfaces (APIs) for desktop and server applications. Previously known as Win32 API.
2,575 questions

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.