[Newsletters Archive ^] [< Volume 1, Number 5] [Volume 2, Number 2 >]

The Systems Internals Newsletter Volume 2, Number 1

http://www.sysinternals.com

Copyright © 2000 Mark Russinovich

January 6, 2000 - In this issue:

1. WHAT'S NEW AT SYSTEMS INTERNALS

   • PsKill v1.0
   • PsList v1.1
   • WinObj v2.1
   • Contig v1.3
   • NTFSCHK v1.0
   • HandleEx v2.1
   • Ctrl2cap v2.0
   • Filemon v4.26
   • Bluescreen v2.1
   • Fundelete v2.01
   • Openlist v1.11
   • December NT Internals

2. INTERNALS NEWS

   • Win2K DDK Released
   • Crash Win2K With a Keystroke
   • Write-protected System Memory Update
   • Win2K API Explosion
   • David Solomon Seminars

3. WHAT'S COMING UP

   • Microsoft NT-Related Patents

SPONSOR: WINTERNALS SOFTWARE

The Systems Internals Newsletter is sponsored by Winternals Software, on the Web at http://www.winternals.com. Winternals Software is the leading developer and provider of advanced systems tools for Windows NT/2K. Winternals Software products include FAT32 for Windows NT 4.0, ERD Commander Professional Edition (advanced boot-disk capability for Windows NT), and Remote Recover.

Winternals Software's NTFSDOS Professional and NTFS for Win98 provide you full read and write access to your NTFS drives from DOS, Windows 95 and Windows 98. NTFSDOS Pro brings single-floppy "boot-disk" capability to Windows NT/2K. With NTFSDOS Pro you can delete buggy drivers, refresh files, and perform general file system maintenance on NTFS drives from a DOS boot floppy. NTFS for Win98 gives you transparent access to NTFS drives from Windows 95 and Windows 98. Easily share applications and files between NT and Win9x on NTFS drives in your dual-boot environment. Both utilities even have built-in NTFS Chkdsk capability. A free read-only version of NTFSDOS Pro is available at http://www.sysinternals.com/ntfspro.htm and a free read-only version of NTFS for Win98 is available at http://www.sysinternals.com/ntfs98.htm.

Hello everyone,

Welcome to the Systems Internals newsletter. The newsletter currently has 14,000 subscribers.

As I'm sure you're aware, Win2K is at the disk duplicator. The Release-to-Manufacturing (RTM) version of Win2K ended up being build 2195. RC3 was 2128, and as I described in an earlier newsletter, Microsoft increments the build number every night - weekends and holidays included - when they compile the current source tree.

I was out at Microsoft in November (see the Filemon update later in the newsletter for the reason why I was there) and a member of the kernel team took me on a tour of the Building 26 at Microsoft's campus. Building 26 is where the Windows NT/2K base kernel team is housed and is where the Windows NT/2K build and test labs are located. The build and test labs are roughly equal in size (maybe 30' by 60'), but the test lab is crammed with racks of computers whereas the build lab has desk space and seats for the builders. Every night the builders extract the source tree onto several quad-processor systems and run a compile. If someone's code check-in happens to (God forbid) break the build, that person is called - regardless of the hour to get their butt in to fix the problem. By mid-November Win2K was under a code freeze where the only changes allowed had to be approved by committees and be directed at "show-stopper" bug fixes.

Once a fresh build is produced the testers take it and install it simultaneously onto all the machines in their lab. The test lab's racks of systems are filled with everything from small hand-held-sized computers to dishwasher-sized multiprocessor servers from every significant PC vendor. After Win2K installation finishes the systems run extensive stress-testing scripts. During the later part of its development Win2K passed stress tests at rates higher than ninety percent. The rates were much lower prior to the introduction of several Win2K reliability enhancements like the Driver Verifier, a tool that helps developers catch problems during their own testing.

Regardless of whether or not a build passes all of the tests it is uploaded to Microsoft's internal distribution server where Microsoft employees can download and install it. If a developer has introduced a significant problem they'll look forward to e-mails from the several hundred employees that end up encountering it over the following week. Its only when there is a serious problem sure to hit a large percentage of users that the test team sends corporate-wide e-mail warning the company (broadcasting e-mail to over 25,000 people is something not to be taken lightly).

While I was there I also met Dave Cutler, the chief architect of Windows NT. What's he up to these days? In November the kernel team was already working hard on the successor to Win2K (known internally as NT 6, or Neptune), and Dave was working on touching-up the installation for the 64-bit version of Win2K. Dave led the 64-bit development effort and 64-bit Win2K is well on its way to completion. As of November the kernel team was still doing 64-bit work on Alphas because Intel had only recently begun to produce samples of Merced processors and there was only one on campus.

As usual, please pass the newsletter on to friends that you think might find it interesting.

Thanks!

-Mark

WHAT'S NEW AT SYSTEMS INTERNALS

PSKILL V1.0

The Windows NT and Win2K Resource Kits come with a command-line "kill" utility, but only because Windows NT and Win2K lack one. You can terminate local processes with the Resource Kit "kill" but not remote ones. I decided therefore to write a freely available "kill" that, like my PsList, has remote capability. PsKill takes either a process ID or name, and an optional computer name, and terminates matching processes on either the local system or the remote one that you specify. You don't even have to install a client component on the remote computer. If the account you are running in doesn't have administrative privilege on the remote computer you can login to the remote system to perform the kill by adding an account name and password to the PsKill command line.

Download PsKill v1.0 at http://www.sysinternals.com/pskill.htm.

PSLIST V1.1

I released PsList some time ago as a UNIX ps-style process and thread viewer. Unlike the similar tools in the Windows NT and Win2K Resource Kits, PsList lets you view process and thread information on remote systems as well as local ones. PsList works by reading Win NT/2K's Performance API information like Perfmon does. This PsList revision adds the ability for you to login to a remote system by specifying an account name and password on its command line. This option allows you to access remote computers for which the account you run PsList from does not have administrative privilege.

Download PsList v1.1 at http://www.sysinternals.com/pslist.htm.

WINOBJ V2.1

WinObj is an Object Manager namespace viewer for Windows NT/2K. The Object Manager namespace is a namespace that is generally not visible to users, but is where all named Win32 (\BaseNamedObjects and \??), and named kernel objects reside. It also serves as the entry point to the file system namespaces (via drive letter symbolic links under \??) and the Registry namespace (via the key object \Key).

WinObj is similar to a tool in the Win32 Software Development Kit (SDK) by the same name, but our WinObj does a lot more than the Microsoft version. For instance, when you view an object's properties in our WinObj you'll see reference and handle counts rather than arbitrary numbers (the SDK WinObj has some major bugs). Our WinObj also shows you the state of synchronization objects and object security information.

This latest WinObj update fixes a bug that prevented it from properly displaying some of the long symbolic link values present in Win2K's namespace. It also uses the new more friendly Win2K security editor dialogs when you run it on Win2K (on NT 4 it uses undocumented security editor interfaces supplied by ACLEDIT.DLL). User interface enhancements include recall of what directory you are viewing when you exit so that the next time you start WinObj that directory is selected and the ability for you to sort the directory contents listview window.

Download WinObj v2.1 at http://www.sysinternals.com/winobj.htm.

CONTIG V1.3

Microsoft included a built-in file defragmenting APIs when they released NT 4. I document the APIs and provide sample code that uses the API at http://www.sysinternals.com/defrag.htm. Using the APIs I implemented Contig, a command-line defragmenter that you can use to defragment individual files or directories. Since the initial release of Contig I've received many requests to add a fragmentation analysis option, and I finally got around to implementing it. Contig v1.3 lets you see how fragmented the files you specify have become so that you can determine whether or not you need to perform a more expensive defragmentation process.

Speaking of file defragmentation, Symantec has released the most advanced defragmenter yet, Speedisk 5.0. In order to top the competition it bypasses the defragmenting API and moves blocks around the disk manually so that it can defragment directories and even the MFT while the system is on-line. Contrary to what Executive Software states at http://www.execsoft.com/diskeeper/infosheet.asp#Q9, their Diskeeper product (both version 4.0 and version 5.0) also bypasses the defragmenting API (but their defragmenter is not nearly as advanced as Norton's), specifically when it performs boot-time directory consolidation. Executive's marketing is another lesson in why you can't believe everything you read.

Download Contig v1.3 at http://www.sysinternals.com/contig.htm.

Download PageDefrag, a Registry and paging file defragmeter, at http://www.sysinternals.com/pagedfrg.htm.

NTFSCHK V1.0

A common complaint from power users that install Win2K on their computers alongside NT 4 is that Win2K's automatic upgrade of any NTFS drives to NTFS v5 renders the NT 4 Chkdsk unable to check those drives. Instead of scanning NTFS v5 drives and correcting errors the NT 4 Chkdsk simply announces that it can't run on drives created with newer versions of NTFS and exits. That requires you to boot into Win2K whenever you want to check those drives at least until now.

With NTFSCHK you can run the Win2K version of Chkdsk from NT 4. How? Using the same technology that we developed for executing NT's native Chkdsk from DOS and Windows 9x as part of NTFSDOS Professional and NTFSDOS for Win98, NTFSCHK wraps the Win2K Chkdsk in an environment that looks like Win2K.

Download NTFSCHK v1.0 at http://www.sysinternals.com/ntfschk.htm.

HANDLEEX V2.1

HandleEx is a multifaceted diagnostic utility for Windows NT/2K that shows you what DLLs processes have loaded and what objects they have opened (their handles). HandleEx is useful for tracking down DLL versioning problems, handle leaks, and determining which application is accessing a particular file, directory, or Registry key.

Version 2.1 of HandleEx lets you view the properties of the objects that processes have opened, including reference counts and the state of synchronization objects. You can also view and modify object security attributes using NT's security editors.

Download HandleEx v2.1 at http://www.sysinternals.com/handleex.htm.

CTRL2CAP V2.0

If you've come from a UNIX background then you'll agree with me that the control key on the PC keyboard is in the wrong place: it should be where the caps-lock key is. And who uses the caps-lock key anyway? Ctrl2cap is a keyboard filter driver that changes caps-lock into left-control, removing caps-lock as a side effect (I use the standard left-control as the fire key when I play Half Life).

Although Ctrl2cap v1.0 works on Win2K, using it disables Win2K's power management features something that's a little irritating on laptops. I therefore updated Ctrl2cap to conform to the Windows Driver Model (WDM), which includes being power-management friendly. I supply full source code and the same source files build both the NT 4 and Win2K versions.

Download Ctrl2cap v2.0 with source code at http://www.sysinternals.com/ctrl2cap.htm.

FILEMON V4.26

The reason I was out at Microsoft in November was that Microsoft held a "File System Filter Plugfest" (internally it was called "Irp-olooza"). The plugfest brought together all the major products that are based on Windows NT/2K file system filter drivers, paired them up in a round-robin manner, and ran stress tests against the different pairings. Products represented included around nine different virus scanners, a number of file encryption tools, and a disk quota manager. The goal of the fest was to identify interoperability problems associated with different filter combinations, help find and identify bugs in the major filter products, and maybe even find a bug in Win2K. Since Filemon is one of the most widely used filters in the world, and many of Microsoft's groups rely on it for their development and troubleshooting work, the plugfest organizers invited me to come to the event and represent it.

Filemon passed all the stress tests without incident except one. Since Filemon is a dynamically loaded filter driver it layered above all the products present at the event except one. The product that layered above Filemon is a virus scanner that also dynamically loads it is in fact a product based on Filemon. Since the virus scanner dynamically loads we tried both layering permutations, and in the one where Filemon was on bottom it caused the virus scanner to crash. When Filemon's GUI exited its driver would delete its filter device objects. Its actually illegal for a filter driver to delete a filter device object unless it gets a command from the I/O Manager telling iit to do so (FastIoDetach in file system filters and IRP_MJ_PNP with IRP_MN_REMOVE_DEVICE in WDM). Not surprisingly, the unexpected disappearance of Filemon's device objects caused the virus scanner to access deallocated memory and crash.

Fortunately, Filemon's crash occurred in the last session of the plugfest so I had minimal embarrassment, and since the testing found at least one serious bug or interoperability issue in every product present I was not alone. Filemon v4.26 is the version that corrects the bug discovered at the plugfest.

Even before I attended the plugfest, I found a bug in Filemon that might be of interest to NT device and file system driver developers. I recently modified Filemon to use the poorly documented Executive Resource (E-Resource) synchronization mechanism. Microsoft's file system drivers use E-Resources extensively so I thought that it would be educational to include their use in Filemon's source code. E-Resources must be acquired by threads that have APCs (Asynchronous Procedure Calls) disabled. You just have to "know" this because the DDK docs don't tell you. Unfortunately, in the haste of implementation I omitted required calls to functions that disable and re-enable APCs around Filemon's E-Resource acquisitions. This bug only causes problems in very rare circumstances so I didn't detect it until Win2K's Driver Verifier caught it for me. To fix the problem I added a call to KeEnterCriticalSection before acquiring an E-Resource and KeLeaveCriticalSection after releasing an E-Resource.

Download Filemon v4.26 at http://www.sysinternals.com/filemon.htm.

BLUESCREEN V2.1

The Bluescreen Screen Saver is a screen saver I wrote that simulates the dreaded Windows NT Blue Screen of Death (BSOD). I wrote the original version before Win2K releases were available, so it simulated the NT 4 BSOD and restart, complete with Chkdsk detecting disk errors. I made two versions available: one that performed disk I/O for added realism and one that didn't. After Win2K Beta 3 was out I updated Bluescreen to simulate the new Win2K BSOD and system restart. In RC3 the restart screen changed so I had to update Bluescreen again. At the same time I made the disk I/O generation an option configurable with Bluescreen's screen saver properties instead of having two versions.

Download Bluescreen v2.1 at http://www.sysinternals.com/bluescrn.htm.

FUNDELETE V2.01

After a long, long wait, our Undelete for Windows NT makes its return as Fundelete for Windows NT. Fundelete is a utility that enhances the Windows NT/2K Recycle Bin to capture files deleted from within programs and the command-line as well as those deleted from Explorer. Why the name change? Several months after Bryce and I released Undelete for Windows NT, Executive Software released Network Undelete, a similar utility. A year later they decided they liked the name of our utility better than their own, so they changed theirs to Undelete for Windows NT. At the same time they had their lawyers send us a letter warning us that we were infringing the registered trademark on the word "undelete" that they have held since 1987. We changed the name of our utility rather than fight.

Developers can download source code to the core of Fundelete's device driver, which demonstrates some powerful driver techniques including obtaining a user's SID from a driver, enumerating a directory's contents from a driver, and creating new IRPs.

Download Fundelete for Windows NT v2.01 at http://www.sysinternals.com/fundelete.htm.

OPENLIST V1.11

Openlist is a Windows 9x utility that shows you all the files that are opened on the system. Version 1.11 adds the ability for you to view the detailed information about the files, including version information for DLLs.

Download Openlist v1.11 at http://www.sysinternals.com/openlist.htm.

DECEMBER "NT INTERNALS"

My "NT Internals" column in the December issue of Windows NT Magazine is "Inside Win2K Scalability Enhancements, Part 2". This second in a two-part series describes the enhancements Microsoft has made in Win2K for multiprocessor scalability including the Job object, new quantum controls, new scheduling classes, and user-mode thread pools.

Last August Windows NT Magazine changed their on-line article browsing policy so that only subscribers were allowed access. Last month they relaxed the policy back to about where it was before August. Now non-subscribers can freely view articles that are more than four issues old.

See a complete list of our publications at http://www.sysinternals.com/publ.htm.

INTERNALS NEWS

WIN2K DDK RELEASED

The final release of Microsoft's Win2K Device Driver Development Kit (DDK) is now available at http://www.microsoft.com/ddk. You can download the kit free or browse the documentation on-line.

CRASH WIN2K WITH A KEYSTROKE

No, it's not a bug. David Solomon, the author of "Inside Windows NT 2nd Edition," supplied me with this cool tip. If you add the DWORD Registry value HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\i8042prt\Parameters\CrashOnCtrlScroll, set it to "1" and reboot, you'll be able to crash Win2K using the keyboard. While holding down the right control key press the scroll key twice in succession. On the second press of the scroll key the system will blue screen with the message "The end-user manually generated the crashdump".

Having the ability to manually crash they system is useful in cases where the kernel or device driver has become deadlocked and the computer is no longer responding. A crash dump generated while the deadlock exists can provide developers information that indicates the cause of the deadlock. This option was introduced so quietly that even Win2K's core kernel developers weren't aware of it until I passed it along when I was at the plugfest.

WRITE-PROTECTED SYSTEM MEMORY UPDATE

In a previous newsletter I talked about write-protected system memory as being a new reliability feature in Win2K. As it turns out, full write protection is not present by default in many configurations. If a computer has at least 128MB of physical memory Win2K uses 4MB "large pages" to map kernel memory. Using 4MB instead of 4KB pages saves a level of page translation and therefore improves performance. Because both read-only code and read/write data may reside on the same 4MB page write-protection is disabled on those systems unless the user requests write-protection using the Driver Verifier. If the Driver Verifier enforces write protection then Win2K uses slower 4KB pages to map kernel memory different memory regions are page-aligned, which means that it is okay to mark individual code pages as read-only.

Thus, write protection is only active on systems with less than 128MB of memory and those where Driver Verifier has enabled it. For systems where write-protection is not active Microsoft is considering the inclusion in a Win2K service pack of a watchdog facility that checksums system memory and then periodically verifies memory against the checksum. The verification operation, though not as precise as hardware-assisted write protection, would detect errant writes to areas that should be read-only.

WIN2K API EXPLOSION

Win2K is without question significantly larger than NT 4. Granted, there are many new services and integrated features that are counted as part of Win2K's size (Active Directory, MMC, COM+, etc.), but even the core OS has grown. One reason the size of the OS has increased is that the number of APIs it exports for applications has increased. The Win2K core OS DLLs include KERNEL32.DLL, GDI32.DLL, USER32.DLL and ADVAPI32.DLL (NTDLL.DLL is also a core OS DLL, but KERNEL32 relies on NTDLL for Win32 APIs). Let's take a quick look at API explosion in each. Here are the raw numbers:

Note that in some cases the growth is artificially inflated by as much as 30% because some APIs come in both ANSI and wide-string forms and are therefore counted twice in the above numbers.

KERNEL32 is the DLL that exports so-called "base OS" functionality, including process, memory, file I/O, and locale management APIs. The APIs that are new to Win2K include new language functions (e.g. EnumUILanguages), Job Object functions (e.g. AssignProcessToJobObject), memory management functions (e.g. AllocateUserPhysicalPages), file functions (e.g. FindFirstVolume), and ToolHelp32 APIs (e.g. Process32First).

GDI32 supplies drawing and bitmap-related routines. Its growth is due to the appearance of mostly miscellaneous new functions that include new font-management APIs (e.g. CreateFontIndirectEx), alpha blending and path-object functions.

USER32 implements windowing functions and a significant part of its growth is with new multiple-monitor APIs. Other new USER32 APIs include a bunch of informational functions (e.g. GetWindowInfo, GetTitleBarInfo).

Finally, ADVAPI32 is the DLL that supplies advanced Win32 APIs. There are a number of new API groups contributing to its growth: EFS (e.g. DecryptFile), CryptoAPI (e.g. CryptEnumProviders), security (e.g. CheckTokenMembership), event-tracing (e.g. StartTrace), and Windows Management Interface (WMI) (e.g. WmiOpenBlock) make up the bulk of the new functions.

DAVID SOLOMON SEMINARS

David Solomon Expert Seminars comes to San Diego - February 21-25. Developer training by the guys who teach at Microsoft.

• Win32 Programming by Jeffrey Richter
• Power Debugging by John Robbins
• Windows 2000 Device Drivers by Jamie Hanrahan
• Windows CE Device Drivers & Applications by Doug Boling

For details, see http://www.solsem.com

WHAT'S COMING UP

MICROSOFT NT-RELATED PATENTS

Software patenting has become a required pastime for companies wanting to leverage their intellectual property. Microsoft is no stranger to the patent game, and NT's kernel has a few mechanisms that have been deemed worthy by the US Patent and Trademark Office (PTO). Areas of the kernel for which Microsoft has obtained patents include the I/O Manager and the Object Manager. Next time I'll give you a list of the patents I've been able to dig up on the NT kernel.

Thank you for reading the Systems Internals Newsletter.

Published Thursday, January 06, 2000 7:09 PM by ottoh

[Newsletters Archive ^] [< Volume 1, Number 5] [Volume 2, Number 2 >]