Hi, I'm trying to understand why a windows 2019 server running on a Dell R740 with 4x nvme disks in raid 0 (using the dell controller) have read speeds of 10GB/s but struggle to write a couple thousand files a second, and when I profile it seems to be every few seconds I see:
ntfs.sys!NtfsCheckpointVolume
That call seems to be single threaded which is a bit of a shame on a 256 logical cpu machine :)
Is there a way to not have the ntfs subsystem checkpointing the volume all the time? I have the USN journal turned on with this configs:
Usn Journal ID : 0x01d676fc684eaf91
First Usn : 0x0000000000000000
Next Usn : 0x00000000139682d0
Lowest Valid Usn : 0x0000000000000000
Max Usn : 0x7fffffffffff0000
Maximum Size : 0x0000000020000000
Allocation Delta : 0x0000000000400000
Minimum record version supported : 2
Maximum record version supported : 4
Write range tracking: Disabled
...
Read more about it, apparently I can get more info by running fsutil fsinfo statistics d:
File System Type : NTFS
UserFileReads : 7651
UserFileReadBytes : 408943616
UserDiskReads : 9076
UserFileWrites : 14494
UserFileWriteBytes : 189571072
UserDiskWrites : 6101
MetaDataReads : 184782
MetaDataReadBytes : 1487081472
MetaDataDiskReads : 276686
MetaDataWrites : 27191
MetaDataWriteBytes : 223002624
MetaDataDiskWrites : 30373
MftReads : 180018
MftReadBytes : 737382400
MftWrites : 25975
MftWriteBytes : 203608064
Mft2Writes : 2
Mft2WriteBytes : 8192
RootIndexReads : 0
RootIndexReadBytes : 0
RootIndexWrites : 0
RootIndexWriteBytes : 0
BitmapReads : 187
BitmapReadBytes : 196083712
BitmapWrites : 1107
BitmapWriteBytes : 8777728
MftBitmapReads : 9
MftBitmapReadBytes : 561152
MftBitmapWrites : 62
MftBitmapWriteBytes : 299008
UserIndexReads : 93740
UserIndexReadBytes : 455147520
UserIndexWrites : 2794
UserIndexWriteBytes : 14729216
LogFileReads : 3
LogFileReadBytes : 12288
LogFileWrites : 443
LogFileWriteBytes : 1209704448
LogFileFull : 881
DiskResourceFailure : 0
VolumeTrimCount : 146
AvgVolumeTrimTime (ms) : 0
AvgVolumeTrimSize (KB) : 2172
AvgVolumeTrimSpeed (KB/s) : 94536349
VolumeTrimSkippedCount : 0
VolumeTrimSkippedSize (KB) : 0
FileLevelTrimCount : 0
AvgFileLevelTrimTime (ms) : 0
AvgFileLevelTrimSize (KB) : 0
AvgFileLevelTrimSpeed (KB/s) : 0
NtfsFillStatInfoFromMftRecordCalledCount : 0
NtfsFillStatInfoFromMftRecordBailedBecauseOfAttributeListCount : 0
NtfsFillStatInfoFromMftRecordBailedBecauseOfNonResReparsePointCount : 0
So there were 881 logfilefull events... this is 10 seconds after starting our application, and it is after changing the log file size unsing chkdsk d: /L:1048576, from the default of 64mb. The results stayed the same if using a 1GB log file or a 64MB log file. How is this possible?
I feel this is related to some way we are accessing files, but I can't understand what might be it.
Another maybe relevant info, all this files are ~ 1kb... so they should all (or 99%) fit in the MFT. Can it be that this LogFileFull are happening because the MFT is growing? It has been set at the default, which IIUC is grow in 200MB increments. I wish I knew how to change that dynamically to 800MB, is that just setting the registry option or do I need to reformat the disk?