[Newsletters Archive ^] [Volume 1, Number 2 >]
The Systems Internals Newsletter Volume 1, Number 1
April 14, 1999 - In this issue:
WHAT'S NEW AT SYSTEMS INTERNALS
- VolumeID for Windows 9x
- EFSDump
- ListDLLs for Compaq Alpha
- "Inside the Boot Process, Part 2"
- My April Windows NT Magazine Article
- No Credit Where its Due
- Not-So-New Stuff
INTERNALS NEWS
- Win2K Driver Verifier
- Y2K Testing with Boot.ini
WHAT'S COMING UP
- Queued spinlocks in Win2K
- Tokenmon
SPONSOR: WINTERNALS SOFTWARE
The Systems Internals Newsletter is sponsored by Winternals Software, on the Web at http://www.winternals.com. Winternals Software is the leading developer and provider of advanced systems tools for Windows NT/2K. Winternals Software products include FAT32 for Windows NT 4.0, ERD Commander (boot-disk capability for Windows NT), and NTRecover.
Hello everyone,
Welcome to the first installment of the Systems Internals Newsletter. I'm pleased to say that the newsletter has already acquired 1000 subscribers since its announcement a little over a week ago.
My goal in the newsletter is to give you timely information on new utilities and articles that are appearing at Systems Internals, plus give you tidbits of information on Windows internals that don't have an appropriate forum at the Systems Internals web site. If you have any comments or suggestions regarding the newsletter, feel free to pass them along to me at mark@.... Also, please forward the newsletter to anyone that you know that might be interested in it. Instructions on subscribing, unsubscribing, or modifying your subscription are provided at the end of the newsletter.
Thanks!
-Mark
WHAT'S NEW AT SYSTEMS INTERNALS
VOLUMEID FOR WINDOWS 9X
While Windows NT and Windows 9x let you change the label on a logical drive or floppy disk using the "label" command, they provide no means for you to be able to change the volume IDs that they arbitrarily assign when you format drives. The VolumeID program, which has been available for Windows NT on the Systems Internals site for over a year, has just been ported to Windows 9x. This applet lets you change the volume IDs on hard drives and floppy disks to whatever you want.
VolumeID uses APIs that allow applications to read and write directly to logical drives, and on Win9x, physical drives (floppy disks), bypassing file systems. On Windows NT/2K VolumeID uses regular ReadFile and WriteFile to access raw drive data - the trick is in how it specifies the name of the volume it wants to access. You can find out how to access physical or logical drives from an application under Windows NT/2K in Microsoft Knowledge Base article Q100027. For Windows 9x access to logical drives, you can base your code on example code provided in Microsoft Knowledge Base article Q174569. If you need to perform direct floppy drive access from your application on Windows 9x, use the Win32 IOCTL VWIN32_DIOC_DOS_INT13, which is documented in MSDN.
Download VolumeID at http://www.sysinternals.com/misc.htm.
EFSDUMP
Windows 2000 will be the debut of Microsoft's built-in file encryption technology, the Encrypting File System. With EFS come a number of new APIs for manipulating encrypted files, including one, QueryUsersOnEncryptedFile, that lets you see which users are registered to have access to encrypted files, and which users are registered as Recovery Agents for those files. I've written a little program called EFSDump that allows you to easily see this information for encrypted files on your system.
While I'm on the subject of EFS APIs, there are number of new APIs that have not been documented as of April's MSDN, something quite disturbing at this late stage in the Windows 2000 release cycle. Most notably, OpenEncryptedFileRaw, ReadFileEncryptedRaw, WriteFileEncryptedRaw, and CloseEncryptedFileRaw (all are exported by Win2K's ADVAPI32.DLL) are all currently undocumented. The use of these APIs is required by any application wishing to backup encrypted files, which means that either Microsoft is passing information about them only to select partners, or backup-software companies are going to have to scramble to get their products out the door when Microsoft eventually does publicly document them. One thing is certain, the developers of Windows 2000's NTBACKUP program have already had access to the API documentation: Win2K Beta 3's NTBACKUP is actively using the APIs.
If you're interested in EFS internals, be sure to check out my upcoming June/July two-part series on EFS in my "NT Internals" column in Windows NT Magazine. In the article I describe exactly where FEKs (File Encryption Keys) and user EFS keys are stored, how the encryption process is performed by NTFS, the EFS driver, and LSASRV (the Local Security Authority Subsystem Server), and of course, how decryption works.
Download EFSDump with full source code at http://www.sysinternals.com/misc.htm.
LISTDLLS FOR COMPAQ ALPHA
The number of utilities available for the Alpha at Systems Internals continues to grow. The latest addition is ListDLLs, a command-line utility that lets you view DLL information for running processes. My HandleEx tool lets you see this information, as well as information on the handles (files, processes, threads, synchronization objects) that processes have opened, but HandleEx is a GUI tool so its not always convenient (it cannot be run inside a batch file for example).
Are you curious about the ratio of downloads of the x86-version of typical Systems Internals utility to its corresponding Alpha version? Its about 20:1, which closely tracks industry estimates of the installed Alpha NT user base as being 5% of the total NT market.
You can download ListDLLs and other Alpha ports, including HandleEx, at http://www.sysinternals.com/alpha.htm.
"INSIDE THE BOOT PROCESS, PART 2"
My January "NT Internals" column from Windows NT Magazine is now available on-line through the Windows NT Magazine web site. You can find a link to it, as well as to Part 1 and older "NT Internals" columns, at the Publications page at Systems Internals: http://www.sysinternals.com/publ.htm.
MY APRIL WINDOWS NT MAGAZINE ARTICLE
Also, be sure to take a look at my feature article in the April issue of Windows NT Magazine, "Linux and the Enterprise". I reveal several significant issues with respect to the Linux 2.2 kernel and network server applications ("enterprise" applications) that will ultimately prevent this version of the Linux kernel from competing head-to-head with NT and other UNIX variants in terms of performance.
NO CREDIT WHERE ITS DUE
On a related note, you may have heard of the recently released D.H. Brown (an analyst company) on Linux's capability as an enterprise operating system. It turns out that the author of the report heavily "borrowed" from an e-mail of mine that someone publicly posted to the Linux kernel Usenet newsgroup back in January. If you have access to the report and read the pages (44 and 45) discussing Linux and multiprocessors ( the report is not publicly available - you must purchase it for a hefty sum), and then read my e-mail at Deja News, you'll see a direct parallel, down to small details.
NOT-SO-NEW STUFF
I'm using this section of the newsletter to bring up recent changes to the Systems Internals site that you may not be aware of, and/or to provide more information about a utility than what is available on the site. For example, a couple of weeks ago we released Filemon v4.1. This version of Filemon is capable of monitoring both Named Pipe and Mail Slot activity under Windows NT/2K. The enhancement to the code that was required to support this was relatively minor, since Named Pipes and Mailslots are implemented as file system drivers. The hard part (tedious) was figuring out the private IOCTLs (I/O Control Commands) that these special file systems support so Filemon can show them. You can download Filemon (which works on Windows NT/2K and Windows 9x) at http://www.sysinternals.com/filemon.htm.
If you would like to learn more about how Filemon works internally, see my February Windows NT Magazine "NT Internals" column, which is entitled "Inside NT Utilities". The article describes how to use Filemon, Regmon, NTFSDOS, NewSID and HandleEx, and tells you a little about how they function.
INTERNALS NEWS
WINDOWS 2000 DRIVER VERIFIER
Windows 2000 Beta 3 is introducing a very powerful device-driver development aid called Driver Verifier. This tool works with code in the kernel to gain control of your driver and exercise it for adherence to kernel-mode rules in a way that was not previously before possible. Buggy device drivers are by far the most significant contribution to the reputation among many people that NT is an unstable operating system, and this tool is aimed at repairing that reputation by helping driver writers find their bugs before users do.
Several types of subtle problems can make it through casual driver testing (the most common type of testing performed under tight time-to-market constraints), and even slip through serious stress tests. One type of common driver problem is a driver accessing paged memory at "elevated IRQL" (high interrupt priority). If the memory the driver accesses happens to be physically present at the time of the access (it hasn't been paged out to the paging file), then the illegal access will go unnoticed. Release a driver that violates this rule out into the wild-world-of-users and its bound to show up, resulting in a Blue Screen crash.
Another type of common driver programming error is for a developer to write the code with the assumption that paged and/or non-paged memory will always be available i.e. they don't check the return values for allocations. If pool runs out and the driver receives a NULL buffer address the driver merrily dereferences the system into a Blue Screen. Although pool exhaustion is a rare occurrence, it's not something that should give a System Administrator a Blue Screen. A related memory bug is a buffer overrun or underrun, where a driver reads or writes outside of a buffer it has allocated.
The Driver Verifier addresses IRQL problems by replacing calls to all functions that manipulate IRQLs in a driver (e.g. KeRaiseIrql, KeAcquireSpinLock) with corresponding verifier kernel functions (VerifierKeRaiseIrql, VerifierKeAcquireSpinLock) at the time the driver loads. Whenever the IRQL is raised to DISPATCH_LEVEL or higher the verifier code calls an internal Memory Manager function, MmTrimAllPageableSystemMemory(), to force all paged data out of physical memory. Thus, the instant that a driver breaks the IRQL rule of not accessing pageable data or code from DISPATCH_LEVEL or higher, the Memory Manager will detect the attempt to access a non-present page and throw a Blue Screen. This lets a developer quickly catch these types of bugs before the driver gets out the door since they can debug the crash and see their driver sitting on the fault stack.
The Driver Verifier accomplishes Memory usage testing with it patching the import table of the driver to be verified so that the driver calls verifier memory functions instead of the standard kernel versions. For example, a call to ExAllocatePool is replaced with a call to VerifierAllocatePool. There are two techniques the Verifier uses to help a developer quickly find memory bugs. The first is that it uses a special pool of memory where a guard page (a guard page is an invalid page) is placed just past the end of the buffer. In addition, the part of the page in which the buffer is allocated that precedes the buffer is filled with a signature. Overruns that are within a page beyond the end of a buffer are detected immediately since they will result in illegal page faults on the guard page. Underruns that involve modification to data preceding a buffer are detected by the Verifier when the driver deallocates the memory, since the signature will have changed.
Drivers that always expect non-empty pool will be tricked into generating crashes by the Verifier with the use of its "memory fault injection". You can configure the Verifier to randomly fail a driver's pool allocations.
There are a handful of other error-types the Driver Verifier checks for, including IRP (I/O Request Packet) consistency, and read-only protection of system and driver code pages.
If you are a device driver developer you'll be doing yourself, your company, and the NT community a favor by testing with the Verifier. Note that you should also test your NT 4.0 drivers that are compatible with Win2K as soon as you have access to the Driver Verifier (most developers will get it with MSDN's shipment of Win2K Beta 3 in late April or May).
For more information, see Driver Verifier.
Y2K TESTING WITH BOOT.INI
If you frequently check the Systems Internals Web site then you're probably already aware of the new Win2K undocumented BOOT.INI switch, /YEAR. I didn't mention on the Web site that the switch is also supported by NT 4.0 Service Pack 4. This switch lets you spoof NT and all software on an NT system into thinking that it's a different year. For example, /YEAR=2001 would make the system think it was 2001 instead of 1999. Thus, the switch is ideal for testing for Y2K problems at any level of software, and the advantage of using it instead of manually resetting the BIOS clock (via the Clock applet, for example) is that the change is temporary and only active when you boot to an installation that has the switch present in its BOOT.INI line. This makes it easy to create a special Y2K installation simply by taking an existing boot line from the BOOT.INI, duplicating it, and adding the /YEAR switch.
WHAT'S COMING UP
Expect the next newsletter in a few weeks. The internals tip I'll have for you next time is about Queued Spinlocks, a new type of spinlock that Windows 2000 uses for its global spinlocks. Here are the global spinlocks that exist in Windows 2000:
- KiDispatcherLock: the scheduler database lock
- KiContext-SwapLock: the tread swap lock
- MmPfnLock: the physical page frame database lock
- MmSystemSpaceLock: the kernel-mode address space lock
- CcMasterSpinLock: the Cache Manager's global spinlock
- CcVacbSpinLock: the Cache Manager's mapping-array lock
Instead of using regular kernel spinlocks (KeAcquireSpinLock, KeReleaseSpinLock) for these global locks as NT 4.0 did, the Windows 2000 kernel uses queued spinlocks (KeAcquireQueuedSpinLock, KeReleaseQueuedSpinLock). These locks have some interesting properties that minimize bus activity on SMPs. Next time I'll tell you how queued spinlocks are implemented.
Coming soon at Systems Internals is Tokenmon, yet another monitoring tool. Tokenmon will show you detailed information about all token-related activity on your system, including logins, logouts, use of privileges, and impersonation.
Thank you for reading the Systems Internals Newsletter.
Published Wednesday, April 14, 1999 7:16 PM by ottoh