Winsock Performance Test (Compact 2013)
3/26/2014
The Winsock Performance Test is a client/server network performance test that operates at the Winsock level. The test measures send and receive throughput, packet loss, and round trip time across any network connection that uses a protocol that the Winsock API supports.
The Winsock Performance Test provides a Windows Embedded Compact version and a Microsoft Windows-based desktop version for both the server and client test modules. You can use the Winsock Performance Test in a variety of scenarios. For information about using the Winsock Performance Test to test a network interface, see Testing a Network Interface by Using the Winsock Performance Test. For information about using the Winsock Performance Test to test a gateway, see Testing a Gateway by Using the Winsock Performance Test.
Before running the test, verify the stability of the hardware connection and the network miniport driver. Microsoft recommends that you use a physical network interface to evaluate the performance of the network driver in a run-time image. Do not use the VMini virtual network adapter with the Winsock Performance Test because the results likely do not represent the maximum possible throughput for the run-time image.
Test Prerequisites
Your device must meet the following requirements before you run this test.
Before running the test, verify the stability of the hardware connection and the network miniport driver. Microsoft recommends that you use a physical network interface to evaluate the performance of the network driver in a run-time image. Do not use the VMini virtual network adapter with the Winsock Performance Test because the results likely do not represent the maximum possible throughput for the run-time image.
The following table shows the software requirements for the Windows Embedded Compact powered device or Windows-based desktop computer that is the client for the Winsock Performance Test. For a device, the run-time image must support Winsock and the network infrastructure that you want to test. For Ethernet-based tests, you must set the SYSGEN_NETWORKING variable in the OS design and the variable for the driver that you want to test.
Requirement |
Description |
|---|---|
Tux.exe |
Tux test harness, required for executing the test. |
Kato.dll |
Kato logging engine, required for logging test data. |
Perflog.dll |
Module that contains functions that monitor and log performance. |
Perf_winsock2.dll |
Test library. |
The following table shows the software requirement for a supporting Windows-based desktop computer or a device acting as the server for the Winsock Performance Test.
Requirement |
Description |
|---|---|
Perf_winsockd2.exe |
Server executable file. |
Subtests
The table below lists the subtests included in this test.
SubTest ID |
Description |
|---|---|
1001 |
TCP Send Throughput. Uses the Winsock sendfunction to send data over TCP from the Perf_winsock2.dll client to the server, which receives data by using the Winsock recvfunction. This test case expresses send throughput in kilobytes per second. The test case measures the time required to send an amount of data, not including protocol headers, which is determined by the specified buffer size and the number of iterations. This test ignores the setup and teardown time for the TCP connection when evaluating throughput. The test case measures CPU use at the client. A value of 100 percent indicates that the CPU is fully used. |
1002 |
TCP Recv Throughput. Uses the Winsock recv function to receive data over TCP at the Perf_winsock2.dll client from the server, which sends data by using the Winsock send function. This test case expresses receive throughput in kilobytes per second. The test case measures the time required to receive an amount of data, not including protocol headers, which is determined by the specified buffer size and the number of iterations. This test ignores the setup and teardown time for the TCP connection when evaluating throughput. The test case measures CPU use at the client. A value of 100 percent indicates that the CPU is fully used. |
1003 |
TCP Ping. Instructs the client and server to run a TCP ping test where the Perf_winsock2.dll client uses the Winsock send function to send a TCP packet, or set of packets making up a specified buffer size, to the server. The server replies by sending the same packet to the client. This test case expresses TCP ping delay in milliseconds. The test case measures the average time elapsed between the client sending a packet and receiving a response from the server. This test ignores setup and teardown times for the TCP connection when evaluating delay. |
1004 |
TCP Send Throughput Nagle Off. Runs test case 1001 with the Nagle algorithm disabled. |
1005 |
TCP Recv Throughput Nagle Off. Runs test case 1002 with the Nagle algorithm disabled. |
1006 |
TCP Ping Nagle Off. Runs test case 1003 with the Nagle algorithm disabled. |
1007 |
UDP Send Throughput. Uses the Winsock send function to send data over UDP from the Perf_winsock2.dll client to the server, which receives data by using the Winsock recv function. This test case expresses send throughput in kilobytes per second. The test case measures the time required to send an amount of data, not including protocol headers, which is determined by the specified buffer size and the number of iterations. The test case measures CPU use at the client. A value of 100 percent indicates that the CPU is fully used. |
1008 |
UDP Send Packet Loss. Sends data over user datagram protocol (UDP) from client to server at different rates to discover when packet loss occurs.For each rate, this test case expresses packet loss by showing the percentage of packets sent that were received by the server. The test case measures CPU use at the client. A value of 100 percent indicates that the CPU is fully used. |
1010 |
UDP Recv Packet Loss. Sends data over UDP from server to client at different rates to discover when packet loss occurs.For each rate of data transmission, this test case expresses packet loss by showing the percentage of packets sent that were received by the client. The test case measures CPU use at the client. A value of 100 percent indicates that the CPU is fully used. |
1009 |
UDP Recv Throughput. Varies the send rate from the server to discover the maximum rate at which the client can receive UDP packets with no packet loss.This test case expresses in kilobytes per second the maximum rate at which the client can receive data in UDP packets, not including protocol headers. The test case measures CPU use at the client. A value of 100 percent indicates that the CPU is fully used. |
1011 |
UDP Ping. Instructs the client and server to run a UDP ping test where the Perf_winsock2.dll client uses the Winsock send function to send a UDP packet, or set of packets making up a specified buffer size, to the server. The server replies by sending the same packet to the client.This test case expresses UDP ping delay in milliseconds. The test case measures the average time elapsed between the client sending a packet and receiving a response from the server. |
1012 |
TCP SendRecv Throughput. This test is a combination of tests 1001 and 1002. It simultaneously compares send and receive throughput. For each packet size, this test sends a 128 KB buffer. |
1013 |
UDP SendRecv Throughput. This test is a combination of tests 1007 and 1009. It simultaneously compares send and receive throughput. For each packet size, this test sends a 128 KB buffer. |
2001 |
TCP Send Throughputs for 10%-100% of the max CPU usage. This test case expresses send throughput in Kbps at CPU usages from 10-100% of the max CPU usages with 10% increments . The test case measures the time required to send an amount of data, not including protocol headers, determined by the specified buffer size and number of iterations. TCP connection setup and tear-down times are ignored when evaluating throughput. |
2002 |
TCP Recv Throughputs for 10%-100% of the max CPU usage. This test case expresses receive throughput in Kbps at CPU usages from 10-100% of the max CPU usages with 10% increments. The test case measures the time required to receive an amount of data, not including protocol headers, determined by the specified buffer size and number of iterations. TCP connection setup and tear-down times are ignored when evaluating throughput. |
2003 |
TCP Send Throughputs Nagle Off for 10%-100% of the max CPU usage. Runs test case 2001 with the Nagle algorithm disabled. |
2004 |
TCP Recv Throughputs Nagle Off for 10%-100% of the max CPU usage. Runs test case 2002 with the Nagle algorithm disabled. |
2005 |
TCP Send/Recv Throughputs for 10%-100% of the max CPU usage. This test case expresses send throughput and receive throughput in Kbps at CPU usages from 10-100% of the max CPU usages with 10% increments. The test case measures the time required to send an amount of data and the time required to receive an amount of data, not including protocol headers, determined by the specified buffer size and number of iterations. TCP connection setup and tear-down times are ignored when evaluating throughput. |
2006 |
TCP Send/Recv Throughputs Nagle Off for 10%-100% of the max CPU usage. Runs test case 2005 with the Nagle algorithm disabled. |
Setting Up the Test
Testing a Network Interface by Using the Winsock Performance Test :
You can use the Winsock Performance Test to test any network interface that Winsock supports. Use the following procedure for testing an Ethernet network interface as a model for testing the network interface that you want to test.
The following table shows the hardware requirements for using the Winsock Performance Test to test an Ethernet network interface on a Windows Embedded Compact-based device.
Requirements |
Description |
|---|---|
Windows Embedded Compact-based device with one network card |
A Windows Embedded Compact-based device with one network interface installed.You can optionally install a second network interface over which the Windows Embedded Compact-based device communicates with the development computer running Platform Builder for Windows Embedded Compact 2013 and the Windows Embedded Compact Test Kit. |
Supporting desktop computer with one network card. |
A Windows-based desktop computer that functions as a supporting desktop computer. On the supporting desktop computer, install an Ethernet network interface that communicates with the tested network interface on the Windows Embedded Compact-based device. Connect the Ethernet network interface on the Windows Embedded Compact-based device to a switch to which you also connect the Ethernet network interface for the supporting desktop computer. If the tested interface is a 100-megabits per second (Mbps) adapter, use a 100-Mbps switch and use a 100-Mbps adapter on the supporting desktop computer. Alternatively, use an Ethernet crossover cable to connect the Ethernet interface on the Windows Embedded Compact-based device to the Ethernet interface for the supporting desktop computer. Optionally, you can use the development computer on which Platform Builder is installed as the supporting desktop computer. Install two network interfaces on the development computer. One network interface is the supporting network interface that communicates with the Windows Embedded Compact-based device. The second network interface is the interface over which Platform Builder and the CTK communicate with the Windows Embedded Compact-based device. |
Ethernet hub or switch, or Ethernet crossover cable. |
If you use a hub or switch, connect to the hub or switch only the target device being tested, the supporting desktop computer, and any other device required by the test. Test results can be significantly affected by sharing the network connection with devices other than those devices used in the test. |
To test a network interface by using the Winsock Performance Test:
* Copy the Perf_winsockd2.exe file from <Platform Builder installation path>\Cepb\Wcetk\Ddtk\Desktop to the supporting desktop computer.
* On the supporting desktop computer, run the following command:perf_winsockd2 -debug .
* On the Windows Embedded Compact-based device, run the desired test for testing the network interface, as listed in Test Cases for the Winsock Performance Test. For example, to run test 1001, use the following command:tux -o -d perf_winsock2 -x 1001 -c"-s server_ip -i ip_version " - OR - If the CTK is connected to the Windows Embedded Compact-based device, in the CTK window append -x 1001 -c"-s server_ip -i ip_version" to the default command line for the test, then run the test.
Note:If desired, you can append -q udp_recv_queue to the above commands to set the UDP receive queue size.
Testing a Gateway by Using the Winsock Performance Test :
The following table shows the hardware requirements for using the Winsock Performance Test to test the throughput of a Windows Embedded Compact-based gateway.
Requirement Description
Windows Embedded Compact-based gateway with two network cards. A Windows Embedded Compact-based gateway with two network interfaces installed.One network interface connects to a private LAN, and the other network interface to a public WAN.You can optionally install a third network interface over which the Windows Embedded Compact-based device communicates with the development computer running Platform Builder and the Windows Embedded Compact Test Kit.
Two supporting desktop computers, each with one network card. Two Windows-based desktop computers, each of which functions as a supporting desktop computer. On each supporting desktop computer, install an Ethernet network interface that communicates with a network interface on the Windows Embedded Compact-based gateway. Connect one Ethernet network interface on the Windows Embedded Compact-based gateway to a switch to which you also connect the Ethernet network interface for one supporting desktop computer. Connect the second Ethernet network interface on the Windows Embedded Compact-based gateway to a second switch to which you also connect the second supporting desktop computer. If the tested interface is a 100-megabits per second (Mbps) adapter, use a 100-Mbps switch and use a 100-Mbps adapter on the supporting desktop computer. Alternatively, use an Ethernet crossover cable to connect each Ethernet interface on the Windows Embedded Compact-based device to the Ethernet interface for the supporting desktop computer. You can use the development computer on which Platform Builder is installed as a supporting desktop computer. Install two network interfaces on the development computer. One network interface is the supporting network interface that communicates with the Windows Embedded Compact-based device. The second network interface is the interface over which Platform Builder and the CTK communicate with the Windows Embedded Compact-based device.
Two Ethernet hubs or switches, or two Ethernet crossover cables. If you use a hub or switch, connect to the hub or switch only the device being tested, the supporting desktop computer, and any other device required by the test. Test results can be significantly affected by sharing the network connection with devices other than those devices used in the test.
Before running the test, verify the stability of the hardware link and the network miniport driver. Microsoft recommends that you use a physical network interface to evaluate the performance of the network driver and the gateway. Do not use the VMini virtual network adapter with the Winsock Performance Test because the results generally do not represent the maximum possible throughput for the gateway.
Verify that the supporting desktop computers can exchange data at a rate that is at least as high as the maximum rate at which gateway can route packets. It is best if the desktop computers can exchange data at the maximum rate supported by the network infrastructure when connected with no gateway through a hub, switch, or with an Ethernet crossover cable.
To test a gateway by using the Winsock Performance Test:
*Copy the Perf_winsockd2.exe file from <CTK installation path>\tests\NT to the supporting desktop computer on the WAN side of the gateway, which functions as a server.Typically, you put the server on the WAN side of the gateway so that the client can make the initial connection to the server and initialize the network address translation (NAT) mappings that are required for the server to reply to the client and run the test. If you put the client on the WAN side, the client cannot establish the initial connection unless, on the gateway, you configure NAT with a Perimeter Network, disable the firewall, and so on.
*From <CTK installation path>\tests\NT, copy the following files to the supporting desktop computer on the LAN side of the gateway, which functions as a client:
Tux.exe
Kato.dll
Perflog.dll
Perf_winsock2.dll
*To ensure that TCP throughput is not capped by TCP window size, use the following registry setting to adjust the TCP window size to 64 kilobits (Kb) on both supporting desktop computers:[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters] . TCP window size differs for the Windows Embedded Compact operating system (OS) and a Windows-based desktop OS. TCP window size also differs for some Windows-based desktop OS versions. TCP window size restricts the amount of data that a sender can send to a receiver at one time without waiting for a delayed acknowledgment (ACK). TCP window size that is not sufficiently large for a network path limits the rate at which data can move between peers. The presence of a gateway in the path that connects two peers might introduce sufficient latency for TCP throughput to be capped by the TCP window size rather than the routing speed of the gateway.
*On the desktop computer on the WAN side of the gateway, run the following command:perf_winsockd2 -debug .
*On the desktop computer on the LAN side of the gateway, run the following command:tux -o -d perf_winsock2 -x test_case -c"-s server_ip -i ip_version ".
Running the Test
Command Line Parameters for the Winsock Performance Test :
The following table shows the command line parameters for the Winsock Performance Test. Before the test can be run, the server_ip address and ip_version must be modifed in the test's command line.
Command line parameter Description
-? Displays a list of command line parameters for the test.
-s server Specifies the server name or server Internet Protocol (IP) address. The default value is localhost.
-i ip_version Specifies which IP version to use. This value must be 4 or 6. The default value is 4.
-b bufsize1,bufsize2,... Specifies a comma-delimited list of buffer sizes through which to iterate.
-n num_buffers Specifies the number of buffers to send or receive. The default value is 10000.
-p num_ping_buffers Specifies the number of buffers to send or receive in ping tests. The default value is 100.
-r send_rate Specifies the rate at which the server or client sends data, in kilobits per second (Kbps). This parameter specifies how often the server or client calls the send function. The default value is max.
-t recv_rate Specifies the rate at which the server or client receives data, in Kbps. This parameter specifies how often the server or client calls the recv function. The default value is max.
-g gw_option Throttles send operations on the client during test case 1007. The default value is 0, which means that throttling is disabled.You can use this parameter when you test a gateway.
-x extension Applies a file name extension to the file name generated by Perflog.dll.
-k ce_priority Sets the priority of the thread that processes send and receive operations. The default value is 251.
-m montecarlo_opt Enable or disables Monte Carlo logging while the test runs. The default value is FALSE.
-c alt_cpu_mon_opt If the run-time image does not support the GetIdleTime function, specifies that the test can use a different method to take CPU measurements. The default value is FALSE.
-q udp_recv_queue Set the byte size of the UDP receive queue. The default is 32768 bytes.
-o cpu_utilCPU consumption between 10% and 100% of the max CPU consumption reached in which the TCP send/recv rates will be measured. E.g.: -o 10 for 10%. The option applies to TCP send/recv tests 1001, 1002, 1004, and 1005. Do NOT use send/recv rate options when this option is used.
-e cpu_utilAdditional CPU utilization between 10% and 90% to be loaded to the system while running the test. This option is applicable to TCP/UDP send/recv tests 1001, 1002, 1004, 1005, 1007, and 1009.
-f <output_file_name>Redirect debug output to file.
-u <display_result_UI>Display a UI with results (default=%s).
-a <gather_net_stats>Gather Networking statistics (default=%s).
-y <Interface IP>Use this interface for Data transfer.
-v <KeepAlive>Set keep alive option for control channel only (default=%s)
.-l <NORlease>Enable current directory, TRUE enables current directory,FALSE enables release directory.
-z <pass_rate_list>Allows listing of comma-delimited pass rate to iterate through, where <pass_rate_list>=<bufSize_PassBandwidthRate_1>,<bufSize_PassBandwidthRate_2>,...,<bufSize_PassBandwidthRate_N>.
Test Cases for the Winsock Performance Test :
By default, each test case for the Winsock Performance Test runs once for each buffer or packet size specified in a predefined list of buffer or packet sizes. For TCP, the following buffer or packet sizes are tested: 16, 32, 64, 128, 256, 512, 1024, 1460, 2048, 2920, 4096, and 8192 bytes. For User Datagram Protocol (UDP), the following buffer or packet sizes are tested: 16, 32, 64, 128, 256, 512, 1024, and 1460 bytes.
By default, a test case sends each packet or buffer 10,000 times for throughput tests and 100 times for ping tests.
Tux -o -d perf_winsock2 -c"-s server_ip -i ip_version -l true"
Verifying the Test
Viewing Test Results for the Winsock Performance Test:
On the Windows Embedded Compact-based device or Windows-based desktop computer that functions as the client, the Winsock Performance Test generates a .txt file for Kato and a .log file. The .log file is a Perflog.dll log file that contains performance data that you can export to .csv format by using the Pparse.exe tool. For more information on performance log files, see Managing Performance Results Using PerfLog.
After the test finishes, the log file is available at one of the following locations:
*In the release directory, if there is a kernel independent transport layer (KITL) connection between the development computer and the device.
*In the root directory of the device, if there is no KITL connection and if you set the following registry setting before running the test:
[HKLM\Software\Microsoft\Perf]
To view test results for the Winsock Performance Test:
*Copy the log file to the development computer.
*From <CTK installation path>\tests\NT, copy Pparse.exe to the directory that contains the log file.
*In the directory that contains the log file, run the following command:pparse log_filename parsed_filename .where log_filename is the name of the log file and parsed_filename is the name of the .csv file that you want to create to store the parsed test results.
*In Excel, open the .csv file.
Troubleshooting the Test
This section describes problems that might be experienced with the Winsock Performance Test. For more troubleshooting help, see Troubleshooting the CTK Tests.
Problem Description Resolution
Winsock Performance Test Does Not Run. When you use Tux.exe to run the Perf_winsock2.dll client, an error message appears containing the text Unable to import Library. This error occurs if a file required by the Winsock Performance Test is not present on the device with the Perf_winsock2.dll client.Before you run the Winsock Performance Test, verify that all required files are in the release directory or in the same directory on the device as the Perf_winsock2.dll file.
Winsock Performance Test Does Not Provide Test Results. When you use Tux.exe to run the Perf_winsock2.dll client, an error message similar to the following error message appears:Couldn't connect to server xxx.xxx.xxx.xxx ,ConnectSocket() for control connection failed, WSAError=10038,Communication between server and client failed; terminating test . This error occurs if communication between the client and server cannot be initiated. Verify that Perf_winsockd2.exe is running on the server. Verify that the server has the IP address specified in the error message. If the server has a different IP address than the IP address specified in the error message, use the -s command line parameter to specify the correct server name or IP address.
Winsock Performance Test Shows UDP Packets Not Received. When you use the Winsock Performance Test to measure the performance of a gateway, UDP-related test cases including the UDP Ping test case indicate that packets fail to cross the gateway or indicate that the round trip time is zero.The Winsock Performance Test relies on port numbers that are not translated. If a device, such as a gateway, uses network address translation (NAT), a TCP or User Datagram Protocol (UDP) port on the private side of the gateway is translated when one of the following conditions is met:Another client uses the port number for the port.To avoid conflicts with other clients, on the test network do not run anything other than the hardware and software required by the Winsock Performance Test.The port number is outside of the range of port numbers that are not translated.For a Windows Embedded Compact-based gateway, the default range of port numbers that are not translated is 1025 to 3000, inclusive. The Winsock Performance Test expects ports in the range of 1024 to 5000, inclusive, to be not translated. If you are testing a Windows Embedded Compact-based gateway, use the following registry settings to extend the reserved port range. [HKEY_LOCAL_MACHINE\Comm\IPNat] "ReservedPortsStart"=DWORD:400 ReservedPortsEnd"=DWORD:1388
Winsock Performance Test Reports Packet Loss despite CPU Cycle Availability Any of the UDP Receive tests report high packet loss, but CPU utilization is lower than 100%. A bug in a miniport driver is likely to lead to this condition. Such a bug will often be echoed by low throughput figures seen in TCP send/receive tests, caused by many retransmissions in the network traffic for the test connection.This may also arise when UDP socket buffer is not of sufficient size. In the UDP receive tests, the rate at which packets arrive at the miniport breaks down to a series of bursts with a delay between each burst. If the receiver cannot completely handle the packets arriving during a burst and its socket queue is not sufficiently large, then packets will be dropped. In this case, the low CPU utilization will be a result of only minimal work done to process small numbers of packets in the queue during the delay between each burst. To increase CPU utilization and increase the receive rate, the receive socket queue can be adjusted with the -q command line test parameter.