[Newsletters Archive ^] [< Volume 2, Number 2] [Volume 2, Number 4 >]

The Systems Internals Newsletter Volume 2, Number 3

http://www.sysinternals.com
Copyright © 2000 Mark Russinovich

June 14, 2000 - In this issue:

1. EDITORIAL

2. WHAT'S NEW AT SYSINTERNALS

   • Regmon v4.25
   • ListDlls v2.22
   • TDImon v1.0
   • AutoRuns v1.1
   • LDMDump v1.0
   • April/June Internals Columns

3. INTERNALS INFORMATION

   • Windows NT Build History
   • Windows NT/2000 Timer Resolution
   • Remapping the Keyboard
   • Safe System Memory Mapping
   • Hidden Windows 98 File System Logging
   • WinDev '00 West

4. WHAT'S COMING UP

   • "Secure" Windows 98 Registry Keys

SPONSOR: WINTERNALS SOFTWARE

The Systems Internals Newsletter is sponsored by Winternals Software, on the Web at http://www.winternals.com. Winternals Software is the leading developer and provider of advanced systems tools for Windows NT/2K. Winternals Software products include FAT32 for Windows NT 4.0, ERD Commander Professional Edition (advanced boot-disk capability for Windows NT), and Remote Recover.

The newly released TCPView Pro allows you to monitor TCP/IP activity on Windows NT 4.0, Windows 2000, and Windows 95/98 systems. Unlike built-in TCP/IP monitoring tools that come with Windows (such as netstat), TCPView Pro shows you which process is associated with each TCP/IP address, making it easy to determine what application is responsible for specific connections and activity. TCPView Pro provides a dynamic view and a static view. The static view shows currently opened local IP addresses, the process associated with each endpoint, and the remote IP address to which an endpoint is connected. The dynamic view, not available with any other utility, allows you to see TCP/IP activity by process in real-time.

Get pricing information and download a 14-day trial version at http://www.winternals.com/products/tcpview.shtml.

Hello everyone,

Welcome to the Systems Internals newsletter. The newsletter currently has 22,000 subscribers.

Dave Solomon and I are in the final stages of wrapping up "Inside Windows 2000, 3rd Ed.", which means the book will be available in mid-August rather than late July (it wouldn't be a Microsoft product without a slip in the ship date). Now that the book is in finalized form, I can give you a rundown on what's in it. First, it has about 50% more content than the previous edition, and includes four brand new chapters. Here's the table of contents:

1. Introduction
2. Architecture
3. System Mechanisms
4. Startup and Shutdown
5. Management Mechanisms
6. Processes and Threads
7. Memory Management
8. Security
9. I/O System
10. Storage
11. Cache Manager
12. File Systems
13. Networking

Like the 2nd edition, the book is full of experiments that demonstrate the concepts we describe. The book also includes a CD that has a copy of the entire SysInternals web site, plus a handful of tools we use in experiments.

Two tools I wrote specifically for the book have been very well received by the book reviewers. The first is named LiveKD and it lets you run any of the Windows 2000 kernel debuggers (i386kd, kd, WinDbg) on a live system. What this means is that you launch LiveKd, specifying which debugger you want it to host, and then you enter the debugger and have available all of the debugger commands that you would if you were debugging a crash dump. Virtually all of the debugger-based experiments in the book can be run using LiveKD, which means that you don't need a second system or a serial cable to perform them.

The second tool is a performance monitor extension that lets you view the live values of any kernel variable. If you wanted to monitor the amount of nonpaged pool in use with PerfMon, for instance, you'd select the MmAllocatedNonPagedPool variable.

I'll let you know in the newsletter when the book is out, but you can preorder now through the Amazon.com link at www.sysinternals.com/links.htm. As usual, please pass the newsletter on to friends that you think would find it interesting.

Thanks!

-Mark

WHAT'S NEW AT SYSTEMS INTERNALS

REGMON V4.25

This latest update to the Regmon Registry monitoring tool includes support for Windows 2000's new KeyNameInformation query type for the ZwEnumerateKey and ZwQuerykey system services. This functionality isn't exported for use by Win32 applications, but is used by the Registry functions in ADVAPI32 as part of the system's use of per-user Class Registration Registry hives.

There are two ways Win32 applications on Windows 2000 can open the Class Registration part of the Registry: they can specify HKEY_CLASS_ROOT or they can specify HKLM\Software\Classes. The first returns a handle to the per-user class key combined with the global class key, and the second returns a handle to the global information only. The ADVAPI32 Registry function can only determine which one the user has specified by examining the underlying name of the Registry key handle passed in by a user, hence the requirement for the new query type. See the SDK documentation on RegOpenKeyEx for more information.

Download Regmon v4.25 at http:www.sysinternals.com/regmon.htm.

LISTDLLS V2.22

When a developer creates a dynamic link library (DLL), they tell the linker the "base address" of the DLL, which is the address for which the linker creates relative address information in the DLL's image file. If a DLL loads at an address different from its base address the loader must fixup all the relative addresses in the loaded DLL image to account for the difference.

These fixups, or relocations, can increase the startup time of an application, so developers obviously want to prevent relocations from occurring. However, it's tedious to look through the output of a program like ListDLLs, comparing load addresses with base address. I've therefore made version 2.22 of ListDLLs take a new option, -r, that has it note relocated DLLs in its output.

Download ListDLLs v2.22 at http://www.sysinternals.com/listdlls.htm.

TDIMON V1.0

TDImon is the latest in the powerful SysInternals monitoring tools suite, showing you TCP and UDP activity on your system as it takes place. The tool takes its name from the fact that it monitors TCP and UDP activity at the interface to the TCP/IP stack, and that interface is called the Transport Driver Interface (TDI). All application and driver TCP and UDP activity must go through this interface, which means that no TCP or UDP activity slips by TDImon undetected.

TDIMon shares the same GUI as its cousins, Filemon, Regmon, Portmon, and DebugView, and like those other monitoring tools it shows you the names of processes performing activity, timestamps, and has filtering and highlighting capability. This makes TDIMon an ideal network troubleshooting tool for administrators, and TCP/IP debugging tool for application developers. TDImon works on Windows 95, 98, NT 4 and Windows 2000.

Download TDImon v1.0 at http://www.sysinternals.com/tdimon.htm.

LDMDUMP V1.0

Windows 2000 includes a new partitioning format called soft partitioning that overcomes some of the drawbacks of the MS-DOS style partitioning that all Windows operating systems have used up until now. A component called the Logical Disk Manager (LDM) manages volumes on disks formatted with soft-partitions, which are called dynamic disks (disks with MS-DOS style partitioning are called basic disks). Besides being more robust because of the partition mirroring they implement, dynamic disks have the advantage that you can create multi-partition volumes without having to reboot the system for them to be recognized and mounted by file system drivers.

Microsoft hasn't documented the format of the LDM partitioning database - in fact, since they licensed the technology from Veritas, who has used the same database in their UNIX volume management software, licensing agreements may prevent Microsoft from documenting it. There may eventually be a Win32 IOCTL interface to the LDM, but in the meantime I've figured out the format and written a tool called LDMDump that you can use to peer inside a dynamic disk's database. LDMDump presents roughly the same information as the Windows 2000 Resource Kit's DmDiag tool, but LDMDump presents the information in (I believe) a much cleaner way. I don't offer source code for this tool at this time, but if you are interested in licensing it for your own applications please contact me.

Read about the LDM Database in my Windows 2000 Magazine "Inside Storage, Part 2" column at http://www.sysinternals.com/publ.htm.

Download LDMDump v1.0 at http://www.sysinternals.com/ldmdump.htm.

AUTORUNS V1.1

You may already be familiar with AutoRuns, which we released within the last two months. AutoRuns shows the auto-run settings for every location in the Registry and .INI files where such information is specified (or so we thought). User feedback has clued us in to a few locations that AutoRuns was missing, and this latest version now shows them.

Download AutoRuns v1.1 at http://www.sysinternals.com/misc.htm.

JUNE/JULY INTERNALS COLUMNS

Have you ever wondered exactly how Win32 services differ from standard Win32 applications? Or maybe you've been curious about what makes the NT boot or shutdown sequence take so long. I answer these questions and more in my two-part June/July series on Win32 services in Windows 2000 Magazine.

In Part 1 I take you inside the structure of a Win32 service, explaining how they accept commands from client applications. Then I begin describing the Service Control Manager (SCM), which is responsible for managing Win32 services, including their startup and shutdown. In Part 2 I finish my description of the service startup process, which takes place during the system boot, and then tell you how SCM shuts down services. I also take a look at improvements Microsoft made to the SCM in Windows 2000, and take you inside the SrvAny Resource Kit tool.

Windows 2000 Magazine subscribers can read the columns on-line at http://www.sysinternals.com/publ.htm.

INTERNALS INFORMATION

WINDOWS NT BUILD HISTORY

As you've learned from past newsletters, the build number for Windows NT (now Windows 2000) increments every day when the build team generates a new build with the day's code check-ins. Using my old beta and release candidate CDs, as well as the help of others that have been using Windows NT longer than I have, I've compiled a list of the build numbers that correspond to public releases (betas, release candidates and full releases). Note that the dates are the date of the build, not the release date for the build. For example, Win2K's final build, 2195, was made in December, but it was released to the public in February.

WINDOWS NT/2000 TIMER RESOLUTION

While Windows NT/2000 provides services, including QueryPerformanceCounter, that let you measure times down to the resolution of the Pentium cycle counter, its interval timing services have a somewhat lower resolution. In fact, the default timer resolution is the same as the system clock interval, which is 10ms on uniprocessor x86 systems (its usually 7.5ms or 15ms on SMP systems). Applications can use the multimedia timer functions in user-space to increase the resolution to 1ms, but drivers are out in the cold if they want higher resolutions - up until Windows 2000, that is.

Windows 2000 introduces a new DDK function, ExSetTimerResolution, that drivers can use to decrease the system timer's interval to 1ms. Want to know what goes on underneath the hood of the multimedia timers and ExSetTimerResolution? See "Inside Windows NT High Resolution Timers" at http://www.sysinternals.com/timer.htm.

SAFE SYSTEM MEMORY MAPPING

While we're on the topic of new Windows 2000 kernel functions for driver developers, it's worth mentioning MmGetSystemAddressForMdlSafe. In previous versions of Windows NT a driver developer that wanted to obtain a system address space pointer for a user's buffer or piece of physical memory had to pass an MDL (Memory Descriptor List) that described the physical buffer to MmGetSystemAddressForMdl.

Creating a virtual mapping in the system's address space uses resource called a System Page Table Entries (System PTEs), where one System PTE is required for every physical page that is mapped. Unfortunately, System PTEs are limited resources and can run out if drivers are mapping large amounts of memory. What happens when MmGetSystemAddressForMdl can't get the System PTEs it requires? You'd think it would do something helpful like return a NULL as the mapped virtual address. But nooooo, it gives up and blue screens the system. Behavior like that reflects badly on the driver making the request.

Windows 2000's MmGetSystemAddressForMdlSafe does what MmGetSystemAddressForMdl should have done: it returns a NULL if there aren't enough System PTEs to create the mapping for the buffer. Use this function to avoid an embarrassing dump that points at your driver. If you have a driver that runs on NT 4 and Windows 2000 its worth releasing two different versions, one for each platform, so that you can take advantage of this new API when on Windows 2000.

REMAPPING THE KEYBOARD

If you're like me, you started out on a UNIX keyboard where a ctrl key was present on the keyboard at the position occupied on PC keyboards by the caps-lock key. To improve my typing rate, and learn something about device driver development on Windows 9x and Windows NT, one of my first driver projects on both of these operating systems was to implement a keyboard remapping driver. You can find the Windows 9x version at http://www.sysinternals.com/c2cap95.htm and the Windows NT/2K version at http://www.sysinternals.com/ctrl2cap.htm.

On Windows NT/2K there's an alternative to using a keyboard filter driver. By defining scancode remapping entries in the Registry, you can completely reprogram the behavior of the keyboard. In fact, the Windows 2000 Resource Kit includes a tool called RemapKey that lets you swap keys around using a graphical representation of the keyboard. This article at Microsoft's web site talks about the keyboard remapper and how it works: http://www.microsoft.com/HWDEV/input/W2kscan-map.htm. Note that the tool also works on NT 4.

So lets say you don't have the Windows 2000 Resource Kit and would rather not spend the money for it (I recommend that you do, its full of all sorts of cool tools and documentation). If that's the case you can manually remap the keyboard. The Microsoft article I just referenced tells you the format of the Registry key where the keyboard driver looks for remapping codes (HKLM\ SYSTEM\CurrentControlSet\Control\Keyboard Layout\Scancode Map), and this article, also available from Microsoft, tells you the scancodes that correspond to keys: http://www.microsoft.com/hwdev/download/desinit/scancode.zip.

If all you want is to swap caps-lock and control (note that my keyboard filters totally do away with the caps-lock key since I never use it), you can copy the following text (not including the "----" separators) to a file (name it something like swapcaps.reg) and double-click on the file. The settings will be imported into the Registry and after a reboot will take effect.

REGEDIT4

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,03,00,00,00,3a,00,1d,00,1d,00,3a,00,\
  00,00,00,00

If you want to undo the mapping just delete the Scancode Map value from the Registry and reboot.

HIDDEN WINDOWS 98 FILE SYSTEM LOGGING

Have you ever browsed your Windows 98 system directory and noticed a subdirectory named \Windows\Applog? Inside this directory you'll likely find files with names matching those of applications you've recently run and extensions like .LGC and .LGD. Open one of the files in Notepad and you'll clearly see a file system activity trace, complete with file names, offsets, and open and close calls. Is a virus generating these logs, or is it a secret utility included with Windows 98 that reports back to Microsoft your application usage patterns? Neither (if it were either, you'd be reading about this in the trade press, not in the SysInternals newsletter). Its part of the "loads your most frequently used applications up to 36 percent faster" feature of Windows 98.

Because of the Taskmon entry in HKLM\Software\Microsoft\Windows\CurrentVersion\Run, Windows 98 launches a service program during the boot named Taskmon. Taskmon loads a VxD named FioLog (\Windows\System\FioLog.Vxd) to install a file system activity hook so that it can see file usage during application launches. Taskmon monitors the file system activity of all applications perform while they start except for ones listed in HKLM\Software\Microsoft\Windows\CurrentVersion\Taskmon\ExcludeApps. FioLog records application startup file system activity in the Applog directory. The log files it creates start with the extension .LGA. It's not clear how it determines when to delete a log and when to create a new one for an application with a new extension with the last letter incremented. Here's a portion of a sample log file:

{
o da3034d0 d000 "C:\WINDOWS\NOTEPAD.EXE"
R da3034d0 0 40
R da3034d0 80 f8
R da3034d0 80 1c0
R da3034d0 7000 1000
R da3034d0 6000 e00
o da2b2610 156000 "C:\WINDOWS\SYSTEM\SHELL32.DLL"
R da2b2610 83000 1000
o da2b2f40 45110 "C:\WINDOWS\SYSTEM\SHLWAPI.DLL"
R da2b2f40 3c000 1000
R da2b2f40 3c000 1000
...

Lines are divided into four fields: the first is the operation code, where o is open, R is read, and C is close. You won't see a W (for write) since FioLog only records read operations during application startup so that application launch can be optimized. The second field is the internal file pointer. The third and fourth fields must be interpreted according to the line's operation code. If the operation code is R then the third field is the file offset and the fourth field is the length of the read. However, if the operation code is o then the third field is open flags, and the fourth is the name of the file that is opened. In the example trace the open of notepad.exe returns the file pointer da3034d0, which you can see being used in subsequent read operations.

When you launch a defrag operation the Defrag.Exe program executes a program named CvtApLog (\Windows\System\Cvtaplog.exe) to process the log files. CvtApLog uses a DLL named ClusAlgo.Dll (\Windows\System\Clusalgo.dll) to figure out optimal cluster placement given the log files it reads, and records this information in files named \Windows\Applog\Applog.d* that guide the defragmentation process. CvtApLog also generates a file named \Windows\Applog\Optlog.txt that summarizes the application-startup optimizations the log files dictate. Here are the partial contents of an Optlog.txt file:

Program Launch Optimization Log - Created Tue Jun 13 11:42:52 2000

Programs Eligible for Optimization:
Ord Flag ProgName Uses   LastExecDate Program Path                           
1        RUNDLL32 65     2000.06.13   C:\WINDOWS\RUNDLL32.EXE                
2        ATIPTAAB 31     2000.06.13   C:\WINDOWS\SYSTEM\ATIPTAAB.EXE         
3        NOTEPAD  22     2000.06.13   C:\WINDOWS\NOTEPAD.EXE                 
4        PING     9      2000.06.10   C:\WINDOWS\PING.EXE                    
…             
17       IEXPLORE 2      2000.06.01   C:\PROGRAM FILES\INTERNET EXPLORER\IEXPLORE.EXE

Programs Ineligible for Optimization:
Ord Flag ProgName Uses   LastExecDate Program Path                           
18  S    GREP     5      2000.06.13   C:\BIN\GREP.EXE                        
19  S    STRINGS  12     2000.06.13   C:\BIN\STRINGS.EXE                     
20  S    ATI2CWXX 31     2000.06.13   C:\WINDOWS\SYSTEM\ATI2CWXX.EXE         

Control Parameters:
Use app profile        = Yes
Minimum log size    = 1000
Maximum no use days = 90
Maximum apps        = 50

Flags for Ineligible Programs:
S = Log size smaller than <Minimum log size>
U = Program not used for more than <Maximum no use days>
P = No profile for program
E = Associated program no longer exists
D = Log deleted (may be combined with one of the above)

Windows 98's ability to move the parts of files used during the launch of an application to a contiguous area on disk is technology Microsoft licensed from Intel (to see this, execute Defrag.exe manually and you'll the text "Intel Application Launch Accelerator").

WINDEV '00 WEST

WinDev '00 East took place last week to a record attendance of 660 people (that's all the hotel could hold). The speakers present at the conference represent the big names in every area of Windows development, including everyone from the COM-god Don Box to driver experts Jamie Hanrahan and Brian Catlin. My sessions included "Windows 2000 Internals", "Advanced Drivers", "Windows NT/2000 File System Drivers" and "Cluster Server".

If you're sorry you missed out, you're in luck because you get a second chance. WinDev '00 West is being held in Santa Clara, CA from September 11-15, and all the same speakers are going to be there. I'll be giving the same sessions, and like at WinDev East, will be giving away free SysInternals t-shirts to attendees that answer my questions or ask particularly insightful ones. You can find more information at http://www.butrain.com/windev/west/default.htm.

WHAT'S COMING UP

"SECURE" WINDOWS 98 REGISTRY KEYS

While the Windows 98 Registry doesn't support security, Microsoft has implemented a mechanism for defining hidden Registry keys. What application makes use of this stealth technology? Internet Explorer, of course. Next time I'll tell you what keys IE hides and how Windows 98 implements them.

Thank you for reading the Systems Internals Newsletter.

Published Wednesday, June 14, 2000 7:08 PM by ottoh

[Newsletters Archive ^] [< Volume 2, Number 2] [Volume 2, Number 4 >]