[Newsletters Archive ^] [< Volume 1, Number 4] [Volume 2, Number 1 >]
The Systems Internals Newsletter Volume 1, Number 5
http://www.sysinternals.com
Copyright 1999 Mark Russinovich
October 12, 1999 - In this issue:
WHAT'S NEW AT SYSTEMS INTERNALS
- NTFS for Windows 98/NTFSDOS Professional
- DebugView v3.21
- Filemon and Regmon v4.21
- Diskmon v1.1
- Systems Internals at www.microsoft.com
- October "NT Internals"
- Not-So-New Stuff
INTERNALS NEWS
- Win2K RC2 DDK Released
- New Win2K RC Kernel APIs
- Software Development '99 East
- Using DiskEdit
WHAT'S COMING UP
- Win2K API Explosion
SPONSOR: WINTERNALS SOFTWARE
The Systems Internals Newsletter is sponsored by Winternals Software, on the Web at http://www.winternals.com. Winternals Software is the leading developer and provider of advanced systems tools for Windows NT/2K. Winternals Software products include FAT32 for Windows NT 4.0, ERD Commander (boot-disk capability for Windows NT), and NTRecover.
Winternals Software's Remote Recover allows you to access the drives of an unbootable computer via TCP/IP from anywhere in your enterprise. Save time and support dollars by remotely correcting driver, file system, and configuration problems that are keeping NT or Win9x systems off-line. You can even perform remote chkdsk or partitioning operations using Remote Recovery, which makes it a versatile disaster recovery tool. Download a free read-only trial version at http://www.sysinternals.com/rrecover.htm, and purchase the read/write version at http://www.winternals.com.
Hello everyone,
Welcome to the Systems Internals newsletter. The newsletter currently has 10,000 subscribers.
The release of Windows 2000 (Win2K) seems to follow a pattern of becoming imminent and then being pushed back. The latest rumors are that it will become available in February. And the only information about the Win2K ship date is in rumors, since Microsoft isn't even telling OEMs or partners when it will ship. Well, they are: "it will ship when it's ready" (I won't force the dated saying about selling wine on you here).
The irritating thing about Microsoft is that the press they generate about their products always makes it seem as if they are on the brink of shipping even when the products are vaporware. The most recent example is Millennium, the successor to Windows 98. Microsoft not too long ago made a big push in the press for it as if it were a finished product, ready to convert all your household appliances into intelligent devices. The irony is that articles a short time later revealed that Microsoft hasn't even fully defined the product yet, and its probably going to be a year or more before we see it on store shelves.
This pattern isn't new and I'm not the first to write about it. But that doesn't stop me from speculating as to how much of the see-sawing is a carefully orchestrated plan, and how much is the result of a total lack of software engineering principals. If you buy into the former angle, as conspiracy theorists do, Microsoft brilliantly stifles competition by tantalizing customers with that incredible product that, if they wait just a little longer, will make their wait worthwhile and obviate any need to turn to a non-Microsoft product. The latter angle says that Microsoft will never learn the software development process, and continuously underestimates engineering effort and overestimates the quality of beta code.
I'm sitting on the fence as to which theory I ascribe to. I actually think the pattern is due to a little of both. I think that it has helped Microsoft to act like Active Directory has been almost ready for two years now. Surely there are customers that would have turned to Novell Directory Services a long time ago if they had known ahead of time how long they would have to wait for Win2K. On the other hand, Microsoft has gotten repeated black eyes in the press from promising product delivery and then delaying. I think that Microsoft's management just doesn't understand how hard it is to reproduce, isolate and fix that last 5% of bugs.
Speaking of Microsoft's development practices, their pre-release naming scheme is one of the most baffling I've seen. Their Betas are really Alphas and their Release Candidates are actually Betas. And even using these definitions is a stretch: Microsoft has no problem ripping features out of their software in going from one Release Candidate to the next, or even adding new ones. They did that with NT 4.0 and they're doing it with Win2K. That practice certainly doesn't speed up a release cycle.
So will Win2K ship in February? I think that we are seeing the light at the end of the tunnel. After all, there are only a handful of new APIs in RC2…
As a follow-up to the last newsletter where I talked about acronym confusion at Microsoft, reader George Janczuk found another example. In the section titled: "Extending to the Mainframe Transaction-Processing World", the article at http://msdn.microsoft.com/library/backgrnd/html/msdn_windnapps.htm refers to CISC - Complex Instruction Set Computing. It's obvious to anyone familiar with mainframe applications that the intended acronym is CICS - Customer Information Control System. A reversed letter sequence and you're in a totally different technology area.
As usual, please pass the newsletter on to friends that you think might find it interesting.
Thanks!
-Mark
WHAT'S NEW AT SYSTEMS INTERNALS
NTFS FOR WINDOWS 98/NTFSDOS PROFESSIONAL
We've finally done it. Ever since Bryce and I released NTFSDOS 1.0 back in the Spring of 1996 we've been in search of the holy grail of Windows file-system compatibility: read/write access for NTFS from Windows 9x and DOS. We determined a long time ago that reverse engineering the NTFS format and writing a driver for this complex journaling file system would be a difficult and risky proposition. Even if one precisely determines every nuance of the format, an update to the format, like Win2K's NTFS v5.0, requires an entirely new reverse-engineering and development effort. Further, validating the correctness of a file system driver for a format as intricate as that of NTFS is a daunting proposition.
So we cheated.
NTFS for Windows 98 provides full read/write access to NTFS drives from Windows 95 or Windows 98, and NTFSDOS Professional provides full NTFS read/write access from DOS. Neither NTFS for Windows 98 nor NTFSDOS Professional has any knowledge of the NTFS file system format. Rather, they rely on the NTFS driver from a Windows NT or Windows 2000 installation to implement that knowledge. Both utilities use the same code-base that serves as an environment wrapper for the NTFS file system driver. NTFS for Windows 98 provides an interface to the Windows 9x file system API above NTFS, and the Windows 9x disk services below NTFS. The interface NTFS for Windows 98 presents to NTFS looks like NTFS' native Windows NT/2K environment, complete with IRPs, the I/O Manager, the Cache Manager, NT-style disk devices, and other NT/2K subsystems that NTFS interacts with.
NTFSDOS Professional works the same way as NTFS for Windows 98, except that it interfaces NTFS to DOS services above and BIOS Interrupt 13 disk services below. In order to help create the NT/2K-like environment we rely on many functions within NTOSKRNL, the NT/2K kernel file. Both utilities load NTOSKRNL in conjunction with NTFS, so that NTFS can directly use may of the kernel's native services.
We had so much fun building the NTFS environment that we went one step further: we did the same thing with NT's boot-time chkdsk utility, Autochk. NTFSDOS Professional and NTFS for Windows 98 come with a NTFS chkdsk utility named NTFSCHK. NTFSCHK works on the same principal as the NTFS file system wrapper, where it creates an NT-like environment for the Autochk program so that Autochk runs under Windows 95/98 and DOS. The result of this trickery is complete NTFS read/write support for Windows 95/98 and for DOS.
You can download a free read-only version of NTFS for Windows 98 at http://www.sysinternals.com/ntfs98.htm and a free read-only version of NTFSDOS Professional at http://www.sysinternalscom/ntfspro.htm. You can purchase the full read/write versions of both tools at Winternals Software, http://www.winternals.com.
DEBUGVIEW V3.21
DebugView is a debug-output monitor that captures both kernel and Win32 debug output under Windows 95, Windows 98, NT 4.0 and Windows 2000. It has advanced filtering, highlighting, and logging capabilities, and can even capture debug output from a system across a network. Its latest release, 3.21, introduces the ability to monitor 16-bit OutputDebugString output under Windows 95 and Windows 98.
You can download DebugView v3.21 at http://www.sysinternals.com/dbgview.htm.
FILEMON AND REGMON V4.21
Filemon and Regmon are file system and Registry spying utilities for Windows 95, 98, NT 4.0 and Win2K. They continue to evolve with new usability features and with release 4.21 (Filemon and Regmon have synchronized version numbers) they introduce more advanced filtering with recent-filter lists, the ability to define a highlight filter (with custom colors even), customizable font, clipboard support, and more CUI-compatible hot-keys. The driver source code has also been reworked and includes more robust parameter validation in the GUI-interface functions.
Download Filemon v4.21 at http://www.sysinternals.com/filemon.htm and Regmon v4.21 at http://www.sysinternals.com/regmon.htm.
DISKMON V1.1
Diskmon is a tool that monitors and displays logical and physical disk activity. Version 1.1 updates Diskmon to work with Windows 2000 and introduces new usability enhancements. In addition, Diskmon now interprets a large number of disk and mass-storage I/O Control codes, translating their hexadecimal codes into text representations in the Diskmon output window.
Diskmon v1.1 not only functions as a disk monitor, but you can use it as a software-based disk light as well. When you set it in disk-light mode Diskmon minimizes itself into the system tray as a light that is green when there is disk read access and red when there is disk write access.
Download Diskmon v1.1 at http://www.sysinternals.com/diskmon.htm.
SYSTEMS INTERNALS AT WWW.MICROSOFT.COM
Considering the history of Systems Internals (formerly NT Internals), it is great flattery indeed that Microsoft references sysinternals.com as a resource for its customers. There are a growing number of Microsoft Knowledge Base articles that point at Systems Internals utilities. Here are the latest:
Q183060 INFO: Troubleshooting Guide for 80004005 & Other Error Messages http://support.microsoft.com/support/kb/articles/Q183/0/60.ASP
This article describes you can use Filemon to check for the cause of data access errors in OLE DB, ActiveX Data Objects and Remote Data Service.Q196453 - Troubleshooting NTVDM and WOW Startup Errors http://support.microsoft.com/support/kb/articles/Q196/4/53.ASP
This article also points at Filemon as a tool for determining what missing files are causing startup errors for NTVDM (the Win16/DOS compatibility environment in NT).Q216368 - PRB: Access Violation During Application Setup When File in Use http://support.microsoft.com/support/kb/articles/Q216/3/68.ASP
HandleEx and DLLView are recommended tools for determining the cause oferrors during execution of setup programs generated by Visual Basic Setup Wizard and Deployment Wizard.Q232830 - HOWTO: Determine File Handle Ownership http://support.microsoft.com/support/kb/articles/Q232/8/30.ASP
HandleEx again gets the referral in this article that discusses finding out what process has a file or directory opened.
OCTOBER "NT INTERNALS"
My "NT Internals" column in the October issue of Windows NT Magazine is "Inside Win2K Reliability Enhancements, Part 3". This third in a three-part series describes two powerful Win2K features that help developers and administrators pinpoint buggy drivers: write-protected kernel memory and the Driver Verifier.
Russian readers can now read my articles in their native tongue. Head over to the Russian edition of Windows NT Magazine at http://www.osp.ru/win2000/ and check out the first translated NT Internals column, Inside the Boot Process (http://www.osp.ru/win2000/1999/10/59.htm). Thanks to Ivan Rouzanov for letting me know about this.
As of the beginning of August, on-line versions of Windows NT Magazine articles are accessible only to subscribers, and articles are posted on-line with each new issue. If you haven't subscribed, please go through the subscription link at http://www.sysinternals.com/publ.htm to get the Systems Internals subscription discount.
NOT-SO-NEW STUFF
WinObj is a powerful tool for exploring the Windows NT/2K Object namespace. The Object namespace is one of three namespaces in NT/2K: the Object namespace, the Registry namespace, and the filesystem namespace. You get to the Registry and filesystem namespaces via objects in the Object namespace. For example, when a Win32 program opens the Registry key HKEY_LOCAL_MACHINE\Software\Microsoft the ADVAPI32.DLL library transforms the name to \Registry\Machine\Software\Microsoft before calling the kernel service NtCreateKey. If you look at the root of the Object namespace in WinObj you'll see an object of type "key" named Registry. The Registry name matches the first component of the key name and so the NT/2K Object Manager passes the rest of the name, \Machine\Software\Microsoft, to the subsystem that defines the key object. The Configuration Manager kernel subsystem maintains the Registry and key objects, so it parses the rest of the name to locate the desired key.
You can explore the Object namespace and view or set object security properties using WinObj. Download Winobj at http://www.sysinternals.com/winobj.htm. I discuss the Object Manager namespace and WinObj in my October 1997 NT Internals column, "Inside the Object Manager". Follow a link to the on-line version of the column at http://www.sysinternals.com/publ.htm.
INTERNALS NEWS
WIN2K RC2 DDK RELEASED
You can download the Win2K RC2 release of Microsoft's Device Driver Development Kit (DDK) now at http://www.microsoft.com/ddk/ddkRC2.htm. You can download the kit for free or browse the documentation on-line.
NEW WIN2K RC2 KERNEL APIS
Things look to be stabilizing in the latest Win2K kernel. There are only four new, exported kernel APIs in RC3, as opposed to about a dozen that appeared (and another half-dozen that disappeared) going from Beta 3 to RC1. Several of the new functions are somewhat interesting, so I've decided to document them for you. The first is MmGetSystemRoutineAddress, and although it's undocumented its prototype is included in the ntddk.h header file of the RC2 DDK:
NTKERNELAPI
PVOID
MmGetSystemRoutineAddress (
IN PUNICODE_STRING SystemRoutineName
);
Its use is as straightforward as it looks. Pass in the name of a function that resides either in NTOSKRNL.EXE or HAL.DLL and you'll get back its entry-point address. This function is actually useless in the first release of Win2K, but may become very useful down the road, and it's a function Microsoft should have included in the first release of NT (3.1). Like GetProcAddress in user-space for Win32 applications, this function lets a driver dynamically ascertain the availability of an API. If Microsoft adds new APIs to Win2K service packs (and I'm sure they will) drivers can be written to take advantage of the APIs, but also to either fail gracefully on older versions of Win2K or to run in a mode where they don't use the APIs. The key to a driver being able to do this is the ability to check for the presence of the APIs after loading. Without this functionality a driver has to statically link with functions it uses, and if the functions aren't present when the driver loads then the kernel loader reports an ugly error to the user and fails to load the driver.
The second new API is MmGetPhysicalMemoryPages. Again, its prototype is in the RC2 ntddk.h but its not documented. Its prototype is:
NTKERNELAPI
PPHYSICAL_MEMORY_RANGE
MmGetPhysicalMemoryRanges (
VOID
);
Where PHYSICAL_MEMORY_RANGE is:
typedef struct _PHYSICAL_MEMORY_RANGE {
PHYSICAL_ADDRESS BaseAddress;
LARGE_INTEGER NumberOfBytes;
} PHYSICAL_MEMORY_RANGE, *PPHYSICAL_MEMORY_RANGE;
This function returns an array of PHYSICAL_MEMORY_RANGE entries with the end of the array marked by an entry that has 0 for both BaseAddress and NumberOfBytes. Like MmGetSystemRoutineAddress, it's a pretty simple API. It returns to you a description of all the physical memory that Win2K knows about. Win2K supports the addition and removal of memory on the fly with the MmAddPhysicalMemory and MmRemovePhysicalMemory APIs. That motivates the reason for the existence of an API that lets you query memory ranges. MmAddPhysicalMemory was added in RC1 and MmRemovePhysicalMemory in RC2. Both of these functions are also prototyped in ntddk.h.
What are the other new RC2 APIs? PoShutdownBugCheck and RtlInvertRangeList. PoShutdownBugCheck lets you crash the system and perform a power-related action like suspending. It's used in a couple places by the RC2 kernel. Ranges are generic start-end specifications that are user-defined and supported by a number of kernel APIs for managing, sorting, and iterating over them. The Win2K Plug-and-Play resource arbiters use them to track and organize hardware-resource requirements. Even though the range-list APIs are not documented, all their prototypes and structure definitions are included in ntddk.h, so you could presumably use the API to manage your own start-end oriented data.
Stay tuned for more fun with undocumented APIs in subsequent newsletters.
SOFTWARE DEVELOPMENT 99 EAST
The 1999 East Coast edition of Software Development is taking place in Washington D.C. from November 8-12. I'm presenting three talks on the last day: What's New in Windows 2000 For Developers, Inside the Windows NT/2000 Blue Screen, and Inside Windows NT/2000 Networking. In addition, Dave Solomon, author of Inside Windows NT 2nd Edition and a neighbor (he lives a mere 20 minutes from me in, of all places, Danbury, CT), and I are hosting a Windows NT/2K internals round-table. We'll be together to answer your toughest questions about Windows NT/2K internals. Say hello and mention the newsletter and I'll give you a free Systems Internals t-shirt!
Visit the Software Development Web site at http://www.sdexpo.com.
USING DISKEDIT
You might not know it, but there is a disk editor utility for Windows NT in the style of the venerable Norton Disk Edit for DOS. What's more, the utility understands FAT and NTFS and it's free. Microsoft apparently shipped DiskEdit accidentally, a tool that must be an internal debugging tool for their file systems teams, on the Windows NT 4.0 Service Pack 4 CD. DiskEdit has a peculiar interface that would take a small manual to document, but I'm going to get you started with a simple walkthrough. I'll focus on using DiskEdit to unravel the NTFS file system format, since DiskEdit is the only publicly available tool I know of that understands NTFS.
First you need to get DiskEdit from the Service Pack 4 (SP4) CD-ROM. Simply copy it from the \i386 directory to your hard disk. If you want to use DiskEdit under Win2K you'll need to create a private directory for it and copy the following DLLs from a SP4 winnt\system32 directory (or SP4 CD) to the same directory as DiskEdit: IFSUTIL.DLL, ULIB.DLL, UNTFS.DLL and UFAT.DLL. Now you can start DiskEdit.
For this walkthrough create a directory called TEMP at the root of a NTFS drive and create a file called OUT.TXT in that directory by typing the following command in a command-prompt window with TEMP as the current directory: echo hello > out.txt. Select the drive with your new OUT.TXT file in DiskEdit by choosing the File|Open menu item and entering the drive's letter in the Volume Name field of the resulting dialog box. Make sure you include the colon e.g. "d:". Virtually all of DiskEdit's functionality requires that you've opened a drive.
We're going to locate the OUT.TXT file by starting at the root directory of the NTFS drive. Select the menu entry Read|NTFS File Record to open a dialog box that lets you view any MFT record entry just by entering its number. The first 16 MFT record entries of every NTFS drive are reserved and correspond to pre-defined NTFS metadata files. Here are the number assignments (note that DiskEdit interprets all input as hexadecimal):
0: $MFT - MFT
1: $MFTMirr - MFT Mirror (a copy of the first 4 entries of the MFT)
2: $LogFile - NTFS LogFile
3: $Volume - volume information file
4: $AttrDef - the attribute definition file
5: . - the root directory
6: $Bitmap - the volume allocation bitmap file
7: $Boot - the boot sector
8: $BadClus - the bad cluster database file
9: $Secure - new to SP4, the security attribute database
A: $UpCase - the lower-to-upper case mapping file
B: $Extend - new to Win2K, the directory that contains
the reparse, object ID, and quota metadata files
C-F: Unused as of NTFS v5.0 (Win2K)
Go ahead and look at some of these MFT entries. You'll start to notice a common theme: they all consist of attributes like $INDEX_ROOT, $FILE_NAME, and $DATA. It's in attributes where data specific to a file is stored. When attribute data is small NTFS stores the data within the file's MFT record as "resident" data, and when the data is large NTFS stores the data external to the record in clusters on disk as "non-resident" data.
Now enter "5" as the file number and you'll be viewing the root directory's file. We're going to look at the files and directories that are in the root directory by viewing the directory's $INDEX_ALLOCATION attribute, an attribute specific to directories that records a directory's contents. To do so select the Read|NTFS Attribute menu entry, which opens another dialog box. DiskEdit is sensitive to case so enter the following precisely as I've listed it:
Base Frs Number: 5
Base Frs (Base File Record Segment) is another name for MFT number. You enter to 5 to specify that you want to read an attribute from the root directory.
Attribute Type: $INDEX_ALLOCATION
This indicates to DiskEdit that you want to read the directory's content data. I recommend using the pull-down menu to chose the attribute you want since DiskEdit is very picky about the way the attribute type is entered.
Attribute Name: $I30
If you view the $INDEX_ALLOCATION attribute of the root directory you'll see that "$I30" is listed as its name in its "Type code, name" line, so that's what you enter as the attribute name.
Press OK and you'll see a hexadecimal dump of the attribute's contents. We want to see something more intelligible so select the View|NTFS Index Buffer menu entry. You'll be presented with the listing of the directory's contents. Scroll through the listing until you see the entry that has the name "TEMP". If you don't see it, the entry might be located in the root directory's $INDEX_ROOT attribute, an attribute type also associated with directories, and that always has its value stored in the MFT record. Index root entries and allocation entries together form a B+ tree structure storing all of a directory's entries. If you have to view the $INDEX_ROOT attribute just follow the same steps for viewing that attribute as you did for viewing the $INDEX_ALLOCATION attribute. As you scroll through an index buffer you may come upon double-lines of asterisks: these designate the end of one index buffer and the beginning of the next.
Once you've found the TEMP directory's entry make a note of its File Reference (FRS). Select Read|NTFS File Record and enter TEMP's FRS. Now you're looking at the MFT record for the TEMP directory. We want to find the OUT.TXT file, so we'll have to look through TEMP's contents to find it. View the $INDEX_ALLOCATION (or $INDEX_ROOT) attribute of the TEMP directory, switch to viewing the data as a NTFS Index Buffer, and locate the OUT.TXT file. Remember to enter TEMP's FRS as the base FRS number in the attribute selection dialog. If you just created TEMP then it will only have an $INDEX_ROOT (if you mistype something you'll get the pleasure of seeing on of DiskEdit's empty error dialogs).
When you've found OUT.TXT and determined its FRS use Read|NTFS File Record to look at its MFT entry. Scroll down until you find the $DATA attribute. You're now looking at the location of OUT.TXT's data. Since we made a small file, the data is stored in the MFT record. If you try to view OUT.TXT's $DATA attribute using DiskEdit you'll see nothing, since DiskEdit doesn't properly show resident data (one of DiskEdit's many bugs). So, copy a largish (> 2KB) text file to \TEMP\ OUT.TXT. Now you can view the OUT.TXT data in one of two ways: you can examine the start of the data (or all of it if its contiguously stored on disk) by using Read|NTFS Cluster and specifying the first "lcn" value you see in OUT.TXT's $DATA attribute "Extent List"; or you can use Read|NTFS Attribute and enter "$DATA" as the attribute type and nothing (as in don't type anything into that field) as the attribute name.
Extent lists describe the location of an attribute's non-resident data. Each contiguous block of data of up to 16 clusters in length is described by one extent list entry. An extent list entry specifies a virtual cluster number (vcn), logical cluster number (lcn), and run length. A Vcn is the cluster within the file at which the data described by the entry starts. A Lcn designates the logical cluster where the data is stored on disk, and the runlength is the number of bytes of attribute data at that location (remember, DiskEdit is showing you hexadecimal values).
I walked you through the long way of finding the OUT.TXT file's MFT record by showing you how to scan directory contents. There's a shortcut, however: select Crack|NTFS Path and enter TEMP\OUT.TXT. You'll be presented with OUT.TXT's FRS and you can use Read|NTFS File Record to go right to it.
That brings me to the end of my DiskEdit primer. I encourage you to play around with this nifty tool by browsing your FAT or NTFS drives. Its highly unlikely that you'll ever find occasion to use DiskEdit to modify data in order to get your disk out of a jam, but if you're curious about the NTFS on-disk format (the FAT format is well-documented) this is the perfect tool for investigating.
WHAT'S COMING UP
WIN2K API EXPLOSION
Although there are only 4 new exported, kernel-mode APIs that made their debut in RC2, the number of total APIs that Microsoft has introduced since NT 4.0, both in the kernel and in core Win32 DLLs, has exploded. Next month I'll tell you exactly how many new APIs there are and where they have appeared, and unfortunately at the same time give people that believe that Win2K is a bloated monster some great ammunition for their alt.advocacy.linux Usenet rants.
Thank you for reading the Systems Internals Newsletter.
Published Wednesday, October 20, 1999 7:10 PM by ottoh
[Newsletters Archive ^] [< Volume 1, Number 4] [Volume 2, Number 1 >]