File System Performance Tests for Flash (Compact 2013)

3/26/2014

The File System Performance Test measures the performance of a file system driver (FSD) that uses the Win32 application programming interface. You can use this test to obtain throughput metrics for various file system usage scenarios on a storage device block driver that runs under the FAT, ExFAT, TexFAT, or a custom FSD. The test can also measure the performance of the Object Store file system.

To judge the performance data produced by this test, you need a basis for comparison. For instance, you might use a known theoretical throughput maximum for a specific storage device, or you might compare two data sets obtained by running this test on two different devices, FSDs, or block drivers. The following are examples of useful applications of the File System Performance Test:

* Comparing file system I/O to block driver I/O (the latter can be obtained by running the Storage Device Block Driver Performance Test).

* Comparing file system performance on storage devices from different manufacturers.

* Running a performance test every time there is a known change in any of the underlying components in the file system stack.

This test will generate a number of performance metrics. After running the test, some of these tests can be viewed visually via the GraphTool in the Windows Embedded Compact Test Kit (CTK) application. Other tests' results can be viewed in the fsperflog.xml log file that is generated. These tests were not graphed because they generate one dimensional or a single data point.

Test Name

In GraphTool

Description

Throughput Writefile

Yes

Measures file write throughput using varying buffer sizes

Throughput Readfile

Yes

Measures file read throughput using varying buffer sizes

Throughput Writefile Random

Yes

Measures the throughput while writing varying buffer sizes of data at random offsets

Throughput Readfile Random

Yes

Measures the throughput while reading varying buffer sizes of data from random offsets

Throughput Reverse Readfile

Yes

Measures the throughput while reading a file in reverse using varying buffer sizes

Throughput Overwrite

Yes

Measures the throughput to overwrite an existing file with varying buffer sizes

Create Directory Tree

Yes

Measures the directory depth that can be created per second

SetFilePointer and SetEndOfFile

No

Measures the time it takes to create a large file with SetEndOfFile

Seek Speed(read)

No

Measures how long it takes to perform a specific number of seeks followed by file reads

Seek Speed(write)

No

Measures how long it takes to perform a specific number of seeks followed by file writes

Read/Append to Fragmented File

No

Time to read and write to a large fragmented file

Copy Files(from storage to device root)

No

Time to copy a set of files from storage to device root

Copy Files(from device root to storage)

No

Time to copy a set of files from device root to storage

FindFirstFile FindNextFile

No

Time to perform FindFileFirst and FindFileNext on 50/100/200 files of the same type

FindFirstFile FindNextFile Random

No

Time to perform random FindFileFirst and FindFileNext on 50/100/200 files of the same type

Create File CREATE_NEW

No

Time to create files with different depths across different directories

Create File OPEN_EXISTING

No

First create files with different depths across different directories and measure the time to CreateFile with the OPEN_EXISTING attribute

Test Prerequisites

Your device must meet the following requirements before you run this test.

The following table shows the hardware requirement for the File System Performance Test for Flash.

Requirement

Description

Onboard flash memory

The device must have onboard available that can be used for storage.

A storage device is not required if the test is intended to measure the performance of the Object Store.

The following table shows the software requirements for the File System Performance Test for Flash.

Requirement

Description

Tux.exe

Tux Test Harness, required for executing the test.

Kato.dll

Kato logging engine, required for logging test data.

Fsperflog.dll

Test library.

ceperf.dll

Module containing functions that monitor and log performance.

perfscenario.dll

Module containing functions that monitor and log performance.

Note:When you run the File System Performance Test, the CTK temporarily copies files to the root directory of the target device. Before running the test, verify that there is at least 0.5 megabytes (MB) of free space in the root directory of the target device. If there is not sufficient space in the root directory of the target device, the test cannot run.

Subtests

The table below lists the subtests included in this test.

SubTest ID

Description

2001

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 128 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 128 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2002

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 256 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 256 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2003

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 512 bytes using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 512 byte read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2004

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 1KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 1KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2005

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 2KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 2KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2006

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 4KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 4KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2007

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 8KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 8KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2008

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 16KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 16KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2009

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 32KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 32KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2010

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 64KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 64KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2011

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 128KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 128KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2012

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 256KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 256KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2013

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 512KB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 512KB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

2014

Throughput Test. Measures the I/O throughput of writing and reading a 2MB file in data blocks of 1MB using the WriteFile and ReadFile functions. The test will also perform random seeks (using the SetFilePointer function) followed by 1MB read and writes. The test fails if the file cannot be created, or if any of the read or write operations fail.

3001

Measures the speed of creating a deep nested directory tree 10 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created.

3002

Measures the speed of creating a deep nested directory tree 15 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created.

3003

Measures the speed of creating a deep nested directory tree 20 levels deep using the CreateDirectory function. The test fails if any of the directories cannot be created.

3004

Measures the speed of creating a 500MB file by using SetFilePointer and SetEndOfFile functions. The test fails if the file cannot be created, or if either of the operations fail.

3005

Measures the seek speed by performing 1000 random seeks followed by 1 byte reads using the SetFilePointer and ReadFile functions. The test fails if the file cannot be created, or if any of the seeks or reads fail.

3006

Measures the seek speed by performing 1000 random seeks followed by 1 byte writes using the SetFilePointer and WriteFile functions. The test fails if the file cannot be created, or if any of the seeks or writes fail.

4001

Measures the I/O throughput by continuously reading and appending data to a 100MB file using the SetFilePointer, ReadFile, and WriteFile functions. This test is intended to simulate the behavior of a Digital Video Recorder (DVR) device, which often records and plays back data from the same file. The test fails if the file cannot be created, or if any of the seeks, reads, or writes fail. If the storage device has insufficient space (~300MB), the test will skip.

4002

Measures the speed of copying a folder with 100 files from the storage device to the root directory of the device using the FindFirstFile, FindNextFile, and CopyFile functions. The test fails if the directory or files cannot be created, or if the copy process fails. If this test is run on the Object Store rather than a storage device, then the files will be both created and copied on the Object Store.

4003

Measures the speed of copying a folder with 100 files from the root directory to a storage device using the FindFirstFile, FindNextFile, and CopyFile functions. The test fails if the directory or files cannot be created, or if the copy process fails. If this test is run on the Object Store rather than a storage device, then the files will be both created and copied on the Object Store.

5001

Measures the time to perform FindFileFirst and FindFileNext on same type of 50 files.

5002

Measures the time to perform FindFileFirst and FindFileNext on same type of 100 files.

5003

Measures the time to perform FindFileFirst and FindFileNext on same type of 200 files.

6001

Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 50 files

6002

Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 100 files.

6003

Measures the time to perform FindFileFirst and FindFileNext randomly on same type of 200 files.

7001

Measures the time to CreateFile with the CREATE_NEW attribute up to 10 files across different directories and nested directories.

7002

Creates files with different depths with 10 files across different directories and nested directories. Then measures the time to CreateFile with the OPEN_EXISTING attribute.

Setting Up the Test

* The onboard flash memory must be mounted and able to read/write accessible.

* You must modify the default command line of the test to specify the profile name of the flash storage if it is not "FlashDisk." Or use other parameters to specify the existence of the flash memory volume.

Running the Test

The command line for the File System Performance Test for Flash must be modified to point to the onboard flash memory. For example, if the flash memory uses the profile name of "FlashDisk" then use the option, "-p FlashDisk".

The test will generate raw performance data in a log file named, fsperflog.xml, in the release directory.

You can modify the test by editing the command line. The following table shows the command line parameters for the File System Performance Test for Flash.

tux -o -d fsperflog -c "[-p <profile>|-r <path>|-root] [-nb -wt -flush] [-cpu] [-desc <description>]"

Command line parameter

Description

-p <profile>

Specify the storage profile to which to restrict testing, found in the registry under the HKEY_LOCAL_MACHINE\System\StorageManager\Profiles key.

-r <\path>

Specify the directory in which to run the test. If you provide this option, the test only runs in the specified directory and ignores the -p option. This flag is useful for a file system driver that does not use storage profiles such as a network file system driver. The path does not require quotation marks; for example, -r \Storage Card

-root

Specify that the test should run on the root file system. If you provide this option, the test runs in the root directory and ignores the -p and -r options.

-nb

Specify that all the test files should be opened with FILE_FLAG_NO_BUFFERING. This option can be combined with -wt.

-wt

Specify that all the test files should be opened with FILE_FLAG_WRITE_THROUGH. This option can be combined with -nb.

-flush

Include the calls to the CloseHandle and FlushFileBuffers functions in performance timing calculations.

-cpu

Include the CPU usage information in the performance log.

-desc <string>

Include a user-specified description string in the performance log.

-iosize #

The I/O buffer size for test cases (default = 512)

-filesize

The file size for test cases 2001-2014 (default = 2MB)

-rand

For tests that perform random I/O, uses TUX-provided random seed.

Verifying the Test

When the test completes running, verify that "PASS" appears in the test log for all sub-tests. A "PASS" test result indicates that the test ran successfully, but the test engineer must manually evaluate the results for each device. Investigate any warning messages that are present in the test log.

After running the test, you can view the results in two ways:

* CEPerf output log - fsperflog.xml

* GraphTool output view

In GraphTool output view, you will be able to see the graphs for tests which generate a series of data. You will have to determine if the performance numbers reported by the test meet the performance required by the device. If not, additional optimizations may be needed. The tests which are graphed are:

* Throughput Writefile

* Throughput Readfile

* Throughput Writefile Random

* Throughput Readfile Random

* Throughput Reverse Readfile

* Throughput Overwrite

* Create Directory Tree

Troubleshooting the Test

Ensure that the flash memory is available and is formatted with the specific file system that are supported by the device.

See Also

Other Resources

Storage Media - Flash Tests