Slow disk performance (NtfsCheckpointVolume) on a raid 0 of nvme disks

Leonardo Santagada 1 Reputation point
2020-08-20T15:35:46.673+00:00

Hi, I'm trying to understand why a windows 2019 server running on a Dell R740 with 4x nvme disks in raid 0 (using the dell controller) have read speeds of 10GB/s but struggle to write a couple thousand files a second, and when I profile it seems to be every few seconds I see:

ntfs.sys!NtfsCheckpointVolume

That call seems to be single threaded which is a bit of a shame on a 256 logical cpu machine :)

Is there a way to not have the ntfs subsystem checkpointing the volume all the time? I have the USN journal turned on with this configs:

Usn Journal ID   : 0x01d676fc684eaf91
First Usn        : 0x0000000000000000
Next Usn         : 0x00000000139682d0
Lowest Valid Usn : 0x0000000000000000
Max Usn          : 0x7fffffffffff0000
Maximum Size     : 0x0000000020000000
Allocation Delta : 0x0000000000400000
Minimum record version supported : 2
Maximum record version supported : 4
Write range tracking: Disabled

...
Read more about it, apparently I can get more info by running fsutil fsinfo statistics d:

 File System Type :     NTFS

 UserFileReads :        7651
 UserFileReadBytes :    408943616
 UserDiskReads :        9076
 UserFileWrites :       14494
 UserFileWriteBytes :   189571072
 UserDiskWrites :       6101
 MetaDataReads :        184782
 MetaDataReadBytes :    1487081472
 MetaDataDiskReads :    276686
 MetaDataWrites :       27191
 MetaDataWriteBytes :   223002624
 MetaDataDiskWrites :   30373

 MftReads :                      180018
 MftReadBytes :                  737382400
 MftWrites :                     25975
 MftWriteBytes :                 203608064
 Mft2Writes :                    2
 Mft2WriteBytes :                8192
 RootIndexReads :                0
 RootIndexReadBytes :            0
 RootIndexWrites :               0
 RootIndexWriteBytes :           0
 BitmapReads :                   187
 BitmapReadBytes :               196083712
 BitmapWrites :                  1107
 BitmapWriteBytes :              8777728
 MftBitmapReads :                9
 MftBitmapReadBytes :            561152
 MftBitmapWrites :               62
 MftBitmapWriteBytes :           299008
 UserIndexReads :                93740
 UserIndexReadBytes :            455147520
 UserIndexWrites :               2794
 UserIndexWriteBytes :           14729216
 LogFileReads :                  3
 LogFileReadBytes :              12288
 LogFileWrites :                 443
 LogFileWriteBytes :             1209704448
 LogFileFull :                   881
 DiskResourceFailure :           0
 VolumeTrimCount :               146
 AvgVolumeTrimTime (ms) :        0
 AvgVolumeTrimSize (KB) :        2172
 AvgVolumeTrimSpeed (KB/s) :     94536349
 VolumeTrimSkippedCount :        0
 VolumeTrimSkippedSize (KB) :    0
 FileLevelTrimCount :            0
 AvgFileLevelTrimTime (ms) :     0
 AvgFileLevelTrimSize (KB) :     0
 AvgFileLevelTrimSpeed (KB/s) :  0
 NtfsFillStatInfoFromMftRecordCalledCount :                              0
 NtfsFillStatInfoFromMftRecordBailedBecauseOfAttributeListCount :        0
 NtfsFillStatInfoFromMftRecordBailedBecauseOfNonResReparsePointCount :   0

So there were 881 logfilefull events... this is 10 seconds after starting our application, and it is after changing the log file size unsing chkdsk d: /L:1048576, from the default of 64mb. The results stayed the same if using a 1GB log file or a 64MB log file. How is this possible?

I feel this is related to some way we are accessing files, but I can't understand what might be it.

Another maybe relevant info, all this files are ~ 1kb... so they should all (or 99%) fit in the MFT. Can it be that this LogFileFull are happening because the MFT is growing? It has been set at the default, which IIUC is grow in 200MB increments. I wish I knew how to change that dynamically to 800MB, is that just setting the registry option or do I need to reformat the disk?

Windows Server Storage
Windows Server Storage
Windows Server: A family of Microsoft server operating systems that support enterprise-level management, data storage, applications, and communications.Storage: The hardware and software system used to retain data for subsequent retrieval.
650 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. Ian Xue 36,751 Reputation points Microsoft Vendor
    2020-08-21T08:03:51.91+00:00

    Hi,
    NtfsCheckpointVolume is expected to be called periodically to perform a checkpoint on a volume so that NTFS can restore the consistency of the file system when the computer is restarted after a system failure. This cannot be disabled and the usn data seems to be okay. Don't know what's the write speed and your scenario. Is your IO sequential or random?

    You could first run chkdsk /f to fix errors of the file system if any. You could also try to trun off write-cache buffer flushing in Device Manager

    19269-2020-08-21-151029.png

    Best Regards,
    Ian

    ----------

    Please remember to "Accept Answer" and upvote if the reply is helpful.

    0 comments No comments

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.