Event Tracing for Windows and Network Monitor

  • [アーティクル]

Event Tracing for Windows, (ETW), has been around for quite a while now as it was introduced in Windows 2000. It's basically instrumented logging that describes what a component is doing. Conceptually, it’s something like the proverbial printf("here1") concept used by programmers, but it is present in retail builds . When you enable logging in a component the result is an ETL (Event Trace Log) file. What’s new is that that Network Monitor can read any ETL file. And with the supplied parsers many network oriented ETW providers can be decoded.

How ETW Works

The idea was to standardize tracing so that it could be turned on for any component with a consistent interface. Before ETW, it was common practice to place DEBUG statements that would output to the debugger. But this often required checked builds of the binaries, special registry entries, and super secret knowledge that sometimes required code access. Now the consumer/provider model with built in enumeration advances and standardizes logging.

Unfortunately, it's not all perfect. As the OS has evolved, so has the story for collecting the ETW data. But the great news is that in Windows 7, you can collect data by scenarios and even include the raw network traffic data, all using ETW tracing.

Subscribing to a Provider

ETW uses a subscription model. Subscribing tools provide a GUID or provider name to receive the logging from that component. Network Monitor is not a subscriber at this point, so we'll have to use other tools to gather the ETW data.

Various operating systems contain different ways of getting ETW traces, but one that seems to work on most Windows operating systems is Logman.exe (https://technet.microsoft.com/en-us/library/bb490956.aspx). When you run Logman, you supply a GUID which identifies a particular provider. To list the providers you can use the /query command.

 ]]logman query providers
Provider GUID
-------------------------------------------------------------------------------
.NET Common Language Runtime {E13C0D23-CCBC-4E12-931B-D9CC2EEE27E4}
ACPI Driver Trace Provider {DAB01D4D-2D48-477D-B1C3-DAAD0CE6F06B}
Active Directory Domain Services: SAM {8E598056-8993-11D2-819E-0000F875A064}
Active Directory: Kerberos Client {BBA3ADD2-C229-4CDB-AE2B-57EB6966B0C4}
Active Directory: NetLogon {F33959B4-DBEC-11D2-895B-00C04F79AB69}
...
Microsoft-Windows-Winsock-AFD {E53C6823-7BB8-44BB-90DC-3F86090D48A6}
Microsoft-Windows-Winsock-WS2HELP {D5C25F9A-4D47-493E-9184-40DD397A004D}
...

see many providers listed, but we only provide ETW parsers for a subset of these. This is mainly because our focus for this feature is Windows 7 network troubleshooting. As we extend the parser set on https://www.codeplex.com/nmparsers, we will add new support for providers. In fact as we speak we are looking to publish an ETW parser for USB 2.0.

Capturing an ETW Trace with Logman

To capture ETW data, use Logman along with the GUID or provider name and some other parameters. Winsock is the layer that applications use to communicate over TCP. Let's capture all Winsock related ETW events.

logman start my_winsock_log -p Microsoft-Windows-Winsock-AFD -ets

We send it the start command and the name of our session and log file my_winsock_log. We pass it the name of the provider with -p. Alternatively, we could use the {GUID} instead. Finally we pass "-ets" which says to start logging now.

logman stop my_winsock_log -ets

This command will stop the logging, again based on the name you specified when you started. When this command completes you should have a my_winsock_log.etl file that we can be opened with Network Monitor 3.3 or 3.2.

Windows 7 Scenario based ETW Tracing

The story for ETW becomes much more interesting in the Windows 7 and Server2008 R2 world. While Logman still works, Netsh becomes the tool of choice for collecting ETW traces. Also incredibly useful is the addition of tracing by scenario and the ability to capture network traffic via NDIS with ETW tracing. But more on this a little later.

Netsh to Collect an ETW Trace.

Netsh used to be component centric with regards to tracing. But now tracing is a top level command that you can use to start and stop tracing. So here's an example that works like our previous Logman example.

 ]] netsh trace start provider=Microsoft-Windows-Winsock-AFD TraceFile=my_winsock_log2.etl

Trace configuration:

-------------------------------------------------------------------

Status: Running

Trace File: my_winsock_log2

Append: Off

Circular: On

Max Size: 250 MB

Report: Off

]] netsh trace stop

Correlating traces ... done

Generating data collection ... done

...

Netsh and Scenarios

Scenarios are things like "InternetClient" or "AddressAcquisition". A tracing scenario is defined as a collection of selected event providers. Providers are the individual components of the network protocol stack, such as WinSock, TCP/IP, Windows Filtering Platform and Firewall, Wireless LAN Services, or NDIS.

One of the primary goals for improved network tracing is to allow collection of all relevant information in one step, and then easily view all events associated with a specific action across the network stack. Network tracing provides a quick method for collecting information and diagnosing networking issues by logging events from all providers in the scenario, and then correlating these events by activity.  In other words, related events & network packets are grouped together for given activity across different components in the networking stack, from Winsock down to NDIS.

Let's look at a list of possibilities for Windows 7.

 ]] netsh trace show scenarios

Available scenarios (18):

-------------------------------------------------------------------

AddressAcquisition : Troubleshoot address acquisition-related issues

DirectAccess : Troubleshoot DirectAccess related issues

FileSharing : Troubleshoot common file and printer sharing problems

InternetClient : Diagnose web connectivity issues

InternetServer : Troubleshoot server-side web connectivity issues

L2SEC : Troubleshoot layer 2 authentication related issues

LAN : Troubleshoot wired LAN related issues

Layer2 : Troubleshoot layer 2 connectivity related issues

MBN : Troubleshoot mobile broadband related issues

NDIS : Troubleshoot network adapter related issues

NetConnection : Troubleshoot issues with network connections

P2P-Grouping : Troubleshoot Peer-to-Peer Grouping related issues

P2P-PNRP : Troubleshoot Peer Name Resolution Protocol (PNRP) related issues

RemoteAssistance : Troubleshoot Windows Remote Assistance related issues

RPC : Troubleshoot issues related to RPC framework

WCN : Troubleshoot Windows Connect Now related issues

WFP-IPsec : Troubleshoot Windows Filtering Platform and IPsec related issues

WLAN : Troubleshoot wireless LAN related issues

To enable a scenario and stop it you type the following commands:

netsh trace start scenario=InternetClient

netsh trace stop

Again, an ETW trace file is created but now there are multiple providers within the same trace file. This can help you correlate tracing from multiple places as the problem moves from one component to another.

Icing is "capture=yes"

One of the coolest parts of this new tracing in Windows 7 is that you can capture NDIS network traffic using Netsh. By enabling the capture parameter, you capture network traffic. Not only is this useful for correlating to other component events, but it provides another way to get a network capture.

Since Netsh is remote-able, you could also use this to start a capture on another machine! And using the persistent=yes parameter, you can enable logging during boot to troubleshoot those nagging start up issues.

By adding the capture=yes parameter to any scenario or specific provider collection, network traffic is captured, as well. To get a simple trace, use this command:

netsh trace start capture=yes

The resulting trace contains TCP, IPv4, Ethernet, ...etc. just like a trace taken from the network interface. However the data has extra headers on the front. If you have other provider information mixed in, and the latest parsers form https://www.CodePlex.com/nmparsers, a simple filter to show you just the network traffic is as follows:

NDISPacCap_MicrosoftWindowsNDISPacketCapture

We also update the standard filters on https://www.codeplex.com/nmparsers. As new requests and ideas for standard filters come up, we'll add to the current set.

Netsh Reference

Here's a small table to summarize the Netsh commands referenced:

Start

Starts a trace session

Stop

Stops a trace session

Capture=yes

Turns on raw network capture from NDIS

Provider=ProviderName

Enables tracing for a specific provider

Show Providers

Shows a list of providers

Scenario=ScenarioName

Enables tracing for a specific scenario

Show Scenarios

Shows a list of scenarios

Conversations and Built-in Standard Filters

We've extended the idea of conversations to ETW logs. The conversation tree is populated with activities and providers under that which will help you narrow down interesting traffic. We also provide some standard filters to help you do some basic searches. And as usual, we try to propagate the important information to top of the tree.

ETW

And finally the default columns that show up in Network Monitor 3.3 are different when you open an ETW trace. We add a "UT Process Name" column, which stands for Unified Tracing, which displays possibly the process name, as well as the process ID. This replaces the "Process Name" column that shows up when you open capture files.

Where to Go Next?

When assisted support or further troubleshooting is necessary, the improved network trace logs can enable developers and support professionals to quickly isolate network activities and view the combined event data and packet captures in a single file, grouped by a network task and the related network activity.

On MSDN there's a great reference that discusses Win7 troubleshooting with Event Tracing (https://go.microsoft.com/fwlink/?LinkID=145404). This resource explains in greater detail how to troubleshoot problems, create filters, and use conversations with ETW tracing and Network Monitor. And while it's Windows 7 specific, there is still some applicability to older operating systems. And Network Monitor can read any ETL file. While we don't necessarily have parsers for every provider, that doesn't mean a parser can't be written.

Comments

  • Anonymous
    January 01, 2003
    If you are looking to follow this series, be sure to subscribe to my RSS feed at http://feeds.jasongaylord.com/JasonNGaylord

  • Anonymous
    January 01, 2003
    If you are looking to follow this series, be sure to subscribe to my RSS feed at http://feeds.jasongaylord

  • Anonymous
    January 01, 2003
    thanks

  • Anonymous
    January 02, 2014
    Let’s start this new year with a small post on DirectAccess (DirectAccess remain in my good resolutions

  • Anonymous
    May 16, 2015
    Applies to: Windows Server 2012 R2 Windows 8.1 Windows Server 2012 Windows 8.0 Windows Server 2008 R2

  • Anonymous
    June 10, 2015
      Applies to: Windows Server vNext Windows 10 Windows Server 2012 R2 Windows 8.1 Windows Server