• Article

[Newsletters Archive ^] [< Volume 1, Number 3] [Volume 1, Number 5 >]

The Systems Internals Newsletter Volume 1, Number 4

http://www.sysinternals.com
Copyright (C) 1999 Mark Russinovich

August 5, 1999 - In this issue:

  1. WHAT'S NEW AT SYSTEMS INTERNALS

    • Portmon v3.0
    • Frob v1.5
    • ListDLLs v2.1
    • New BOOT.INI Options
    • PsList v1.0
    • August "NT Internals"
    • Not-So-New Stuff

  2. INTERNALS NEWS

    • Correction Re: System File Protection
    • Win2K RC1 DDK Released
    • The Win2K AWE API
    • WinDev '99 West

  3. WHAT'S COMING UP

    • SP4's Undocumented DiskEdit Tool

SPONSOR: WINTERNALS SOFTWARE

The Systems Internals Newsletter is sponsored by Winternals Software, on the Web at http://www.winternals.com. Winternals Software is the leading developer and provider of advanced systems tools for Windows NT/2K. Winternals Software products include FAT32 for Windows NT 4.0, ERD Commander (boot-disk capability for Windows NT), and NTRecover.

Winternals Software announces the release of its latest system recover utility, Remote Recover. Remote Recover allows you to access the drives of an unbootable computer via TCP/IP from anywhere in your enterprise. Save time and support dollars by remotely correcting driver, file system, and configuration problems that are keeping NT or Win9x systems off-line. Download a free read-only trial version at http://www.sysinternals.com/rrecover.htm, and purchase the read/write version at http://www.winternals.com.

Hello everyone,

Welcome to the fourth edition of the Systems Internals newsletter. The newsletter currently has 7000 subscribers.

In the last newsletter I indicated that I would be covering Windows 2000's (Win2K) AWE API. I pointed out that AWE stands for "Address Windowing Extensions" (for newbies to the world of Windows, API stands for "Application Programming Interface"). This page at Microsoft's hardware developer Web site describes it as such: http://www.microsoft.com/hwdev/NTDRIVERS/AWE.htm. However, reader Laxmikant Gunda pointed me to another URL at Microsoft's Web site, http://www.microsoft.com/windows/server/News/fromMS/intelpae.asp, where AWE is designated the "Advanced Windowing Extensions" API. Clearly, "Address" makes more sense than "Advanced" in the context of the API (which I describe later in this newsletter), so it's my guess that some marketing writer had their eyes play a trick on them as they digested raw technical material for the news page.

I've seen this confusion happen before in Microsoft's technical coverage. Most recently, the setup program for the Beta 3 release of the Win2K Installable File Systems (IFS) Kit announced itself on its splash screen as setup for the "Internal File Systems Kit". The ‘I' in IFS has always stood for "Installable", and "Internal" doesn't make much sense here (unless they accidentally released the non-public version of the kit), so I'm guessing that its another case of something getting screwed up in the translation (or the setup coder not having a file systems-team member handy and using their best guess).

What's my point? I don't really have one, other than that sooner or later we're going to have acronym overrun, where developers and marketing people start inadvertently using the same three-letter acronyms to describe different technologies. Someone in Microsoft's IIS (Internet Information Server) group will name a new API called AIE ("Advanced ISAPI Extensions" - ISAPI stands for Internet Server API) and someone in the MTS (Microsoft Transaction Server) group will come up with another AIE, "Atomic Interface Exchange". And I'm sure that one day a marketing person or technical writer is going to be handed a technical paper from the IIS team that refers to "AIE", and not knowing the difference between a database and a file system, is going to guess that it stands for "Advanced Internet Explorer".

I've seen acronym confusion at work other places, too. A recent Windows NT Magazine article discussing serial and parallel ports used the term "COM port" in its text. After the copy editor got through with a final pass and the article was published the phrase had become "COM (Component Object Model) port".

As usual, please pass the newsletter on to friends that you think might find it interesting.

Thanks!

-Mark

WHAT'S NEW AT SYSTEMS INTERNALS

PORTMON V3.0

Portmon has been enhanced in major ways with its 3.0 release. Portmon is a serial and parallel port monitoring application for Windows 95/98/NT/2K. Version 3.0 introduces:

  • advanced filtering and output highlighting capability
  • the ability to monitor serial and parallel port activity on remote systems
  • log-to-file and printing support
  • clipboard integration
  • a setting that lets you specify how much read/write data to log

Download Portmon v3.0 at http://www.sysinternals.com/portmon.htm.

FROB V1.5

Windows NT 4.0 Server provides administrators with no ability to control the lengths of quantums (turns) that the NT thread scheduler gives to threads. On Windows 4.0 Workstation the System applet in the Control Panel has a Performance tab with a slider that lets you designate a quantum boost for foreground threads (the slider is also present on Server, but it does not do anything). Shorter quantums generally result in applications that are more responsive to user input, while long quantums are good for systems dedicated to running non-interactive server applications.

The Frob program from Systems Internals gives you the same finer-grained degree of control over quantum lengths on both Workstation and on Server, allowing you to tune quantums to the workload you run on a system. However, because Frob modifies settings internal to the NT kernel, it must be updated to work with each new Service Pack. Frob v1.5 is now available, and provides compatibility with Service Pack 5.

You can download Frob v1.5 at http://www.sysinternals.com/ntfrob.htm.

LISTDLLS V2.1

ListDLLs is a command-line utility that shows you detailed version information about the DLLs that processes have loaded. ListDLLs extracts version information from the on-disk files that correspond to the file paths of loaded DLLs, and it obtains the pathnames using undocumented interfaces. Sometimes an on-disk DLL file is updated after an application has already loaded it, in which case ListDLLs shows incorrect version information.

ListDLLs v2.1 recognizes when there is a difference between the on-disk and loaded versions of a DLL, flags the difference in its output. When there is a discrepancy ListDLLs prints out the link times of the on-disk file and the loaded file. Link times of executable and DLL files are located in the header of the Portable Executable (PE) file format that NT uses to package them.

Download ListDLLs v2.1 at http://www.sysinternals.com/listdlls.htm.

NEW BOOT.INI OPTIONS

Win2K Beta 3 introduces three new BOOT.INI switches. All three are related to Intel Physical Address Extensions (PAE), a technology Intel introduced with the Pentium Pro to allow x86 systems to address up to 64GB of physical memory. Traditionally, x86 systems can only address 4GB of physical memory, but with PAE and the 450NX chipset, this barrier is broken. Win2K is the first Microsoft operating system that will take advantage of PAE (Sun Solaris 7 and SCO UnixWare 7 already support PAE). There is actually a special build of the Win2K kernel, named ntkrnlpa.exe, which has the support built-in. NTLDR, the Win2K boot loader, is responsible for loading either the standard kernel, ntoskrnl.exe, or the PAE-enabled one, based on whether the system is capable of addressing more than 4GB of memory and has that amount present.

All three new BOOT.INI switches are aimed at debugging device drivers that are designed to work with large memory systems (systems with more than 4GB). The first, /PAE, has NTLDR load the PAE-version of the kernel even if the computer doesn't have more than 4GB of memory present. The second, /NOPAE, forces NTLDR to load the standard kernel. Finally, the /NOLOWMEM switch has the Win2K kernel only use physical memory above 4GB. This forces all physical addresses used by Win2K to require more than 32-bits to represent them, and thus exercises device driver handling of large physical addresses.

Find the most complete list of BOOT.INI switches available at http://www.sysinternals.com/bootini.htm.

PSLIST V1.0

Most flavors of UNIX ship with a command-line tool called ‘ps' that administrators use to list process CPU and memory statistics. NT has an equivalent GUI-based tool, Task Manager, but NT comes with no command-line version. The Windows NT Resource Kit includes two command-line ‘ps'-like tools, pstat and pumon. PsList shows a combination of the information available with pstat and pumon, but has a capability that neither Resource Kit tool has: you can use PsList to query process information on a remote system.

PsList takes several flags that allow you to view process CPU, memory, or thread statistics, and a switch that lets you specify a different NT computer to query. PsList uses NT's built-in Performance Counters to obtain the information it shows, and it works on NT 3.51, 4.0 and Win2K.

Download PsList at http://www.sysinternals.com/pslist.htm.

AUGUST "NT INTERNALS"

My "NT Internals" column in the August issue of Windows NT Magazine is "Inside Win2K Reliability Enhancements, Part 1". This first in a three-part series describes Win2K features aimed at helping administrators get a system going after it has become unbootable. These include "safe-mode" booting and the Recovery Console.

As of the beginning of August, on-line versions of Windows NT Magazine articles are accessible only to subscribers, and articles are posted on-line with each new issue. If you haven't subscribed, please go through the subscription link at http://www.sysinternals.com/publ.htm to get the Systems Internals subscription discount.

NOT-SO-NEW STUFF

If you haven't scared a friend or two with it already, check out the Systems Internals Bluescreen Screen Saver. Version 2.0 displays the Win2K version of the Blue Screen of Death (BSOD) and the Win2K startup sequence when you run it on a Win2K system. I'm proud to say that the Bluescreen Screen Saver has recently been awarded five stars, the highest rating possible, from Ziff-Davis' Software Library.

Download Bluescreen Screen Saver v2.0 at http://www.sysinternals.com/bluescrn.htm.

INTERNALS NEWS

CORRECTION RE: SYSTEM FILE PROTECTION

In the last newsletter I stated that Win2K's System File Protection (SFP) lets applications like Service Packs (SPs) and hot-fixes update system files by calling a function SFP exports named SfcTerminateWatcherThread. While SFP does export that function, SPs and hot-fixes do not use the function when updating system files. Instead, they are able to update system files because they replace existing system files with ones that are digitally signed by Microsoft. SFP detects the updates, but allows them to stick because SFP verifies that the new files are digitally signed. Another correction regarding my SFP coverage is that after Win2K Beta 3 Microsoft changed the name of SFP to Windows File Protection (WFP).

Be sure to check out the September issue of Windows NT Magazine, where you'll find my NT Internals column "Inside Win2K Reliability Enhancements, Part 2", which describes WFP and Microsoft digital file signing in detail.

WIN2K RC1 DDK RELEASED

You can download the Win2K RC1 release of Microsoft's Device Driver Development Kit (DDK) now at http://www.microsoft.com/ddk/ddk2kRC1.htm. You can download the kit for free, or browse the documentation on-line.

THE WIN2K AWE API

I already mentioned the AWE API in the introduction to this newsletter, and referenced a Web page at Microsoft where you can learn more: http://www.microsoft.com/hwdev/NTDRIVERS/AWE.htm. On systems with more than 4GB of physical memory, Win2K's PAE-capable kernel - ntkrnlpa.exe - is able to take advantage of all the computer's physical memory with no modification to applications. Win2K Advanced Server will use up to 8GB of physical memory and Win2K Datacenter Server will use up to 64GB of physical memory.

While each application on a large memory system has at most 2GB of virtual memory at its disposal (3GB if the /3GB BOOT.INI switch is specified), the sum of physical memory assigned to all executing applications can equal the amount of physical memory. In addition, on Win2K the file system cache is assigned a maximum of 960MB of virtual memory, but the amount of cached file data can be much larger physical memory assigned to the cache can exceed 960MB.

The AWE API gives individual applications the ability to directly control physical memory, and more than the 2GB or 3GB limit implied by their virtual address space size. The basic idea behind the AWE API is that an application designates a portion of its virtual address space as a "window" into physical memory. Then it allocates a chunk of physical memory. The upper limit on the amount of physical memory an application can allocate is essentially the amount of physical memory on the system minus any non-paged memory already allocated by the kernel, device drivers, and other applications using the AWE API. When the application wants to access part of the physical memory it has allocated, it maps the memory into its virtual address window. Thus, the amount of physical memory the application can access with a given mapping is limited by the size of the window that it reserved. Finally, when an application is done with the physical memory it simply frees the memory and closes (deallocates) the virtual address window it created.

The APIs that correspond to these actions are exported by kernel32.dll and are as follows:

  • An application calls VirtualAlloc with the MEM_PHYSICAL and MEM_RESERVE flags to create the virtual address window
  • AllocateUserPhysicalPages allocates physical memory for an application
  • An application uses MapUserPhysicalPages to map portions of the physical memory into its window
  • FreeUserPhysicalPages frees the physical memory the application allocated

The ability for applications to directly manipulate multiple GB's of memory is a boon to memory-intensive programs such as database servers, e-mail servers, Web servers, financial analysis and scientific applications.

While the ability to use more than 4GB of physical memory is only permitted on certain versions of Win2K, the AWE API is present on all versions. This means that on a Win2K Professional system with 4GB of memory, for example, the AWE API still gives memory-intensive applications the ability manage more than 2 or 3GB of data in physical memory.

The AWE API has equivalent kernel-mode interfaces on which kernel32.dll bases the user-mode APIs. A driver can allocate physical memory using MmAllocatePagesForMdl, which is the kernel-mode equivalent of AllocateUserPhysicalPages. This function has its prototype in the Win2K DDK ntddk.h file, but is undocumented. Its prototype is:

NTKERNELAPI
PMDL
MmAllocatePagesForMdl (
    IN PHYSICAL_ADDRESS LowAddress,
    IN PHYSICAL_ADDRESS HighAddress,
    IN PHYSICAL_ADDRESS SkipBytes,
    IN ULONG TotalBytes
    );

LowAddress is the lowest acceptable physical address it wants to allocate and HighAddress is the highest. SkipBytes are the number of bytes that the kernel should keep free above LowAddress and below the address at which it starts to allocate physical memory. TotalBytes are the number of bytes that the driver wants to allocate. The return value of the function is a MDL that if non-zero describes the physical memory the kernel has given the driver. To access portions of the memory the driver must create sub-MDLs from the returned MDL that describe appropriate portions of the physical memory. When a driver wants to access physical memory described by a sub-MDL it must map the sub-MDL using MmGetSystemAddressForMdlSafe.

Drivers use the MmFreePagesFromMdl, the kernel-mode equivalent of FreeUserPhysicalPages, to free the physical memory it has allocated with MmAllocatePagesForMdl. This function is also prototyped in ntddk.h:

NTKERNELAPI
VOID
MmFreePagesFromMdl (
    IN PMDL MemoryDescriptorList
    );

Note that a driver is responsible for deallocating the MDL returned by MmAllocatePagesForMdl with a call to ExFreePool, since MmFreePagesFromMdl does not free the MDL.

WINDEV '99 WEST

The 1999 West Coast edition of the premier conference for Windows developers is fast approaching. WinDev '99 West is taking place September 13-18 at the Santa Clara Marriot in California. A number of big names in Windows development are speaking, including Jeff Richter, Jeff Prosise, and Don Box. I'll be there with Jamie Hanrahan and Brian Catlin in the device driver's track. My presentations include a day-long tutorial on NT internals, as well as one on writing Windows NT/2K file system drivers and one on advanced device driver development issues. Come and say hello!

Visit the WinDev West Web page at http://www.butrain.bu.edu/windev/.

WHAT'S COMING UP

NT 4.0 SP4 DISKEDIT

Windows NT 4.0 Service Pack 4 shipped with a nifty utility on its CD that many people might not have noticed: DiskEdit. And it's no wonder: the tool does not install onto the hard disk, and it comes with no documentation. It's almost certain that it is a tool used internally by Microsoft file system developers that was accidentally shipped on the CD. DiskEdit is a treasure for those people that have missed a Norton Utilities Disk Edit-type tool for Windows NT, or just want to explore NTFS and FAT on-disk data structures. Next time I'll provide you some instructions on how to use DiskEdit.

Thank you for reading the Systems Internals Newsletter.

Published Thursday, August 05, 1999 7:13 PM by ottoh

[Newsletters Archive ^] [< Volume 1, Number 3] [Volume 1, Number 5 >]