It's only temporary

  • Article

NT has a whole lot of really cool features that aren’t always obvious without REALLY looking closely at the documentation.

One of my favorite is what I call “temporary” temporary files.

A “temporary” temporary file is one whose storage is never written to disk (in the absence of memory pressure).  It behaves just like a file does (because it’s a file) but the cache manager disables the lazy writer on the file, and the filesystem knows to never commit the pages containing the file’s metadata to disk.

To create a “temporary” file, you call CreateFile specifying FILE_ATTRIBUTE_TEMPORARY | FILE_FLAG_DELETE_ON_CLOSE in the dwFlagsAndAttributes attribute.  This combination of bits acts as a hint to the filesystem that the file data should never be flushed to disk.  In other words, such a file can be created, written to, and read from without the system ever touching the disk.

Why would you ever want such a file?

Well, consider the case of a web server that’s creating content for client applications.  In this case, it’s possible that halfway through rendering the content to be transmitted to the client, you encounter an error.  The problem is that you’ve already sent the 200 OK response to the client, so the client thinks there are no errors.  To fix this, you can render the content to a “temporary” temporary file, and if there is an error rendering the content, you can response with an appropriate error code if the rendering fails.  If the rendering succeeds, you can use TransmitFile (if your server is written to raw sockets) or HttpSendHttpResponse (if your server is written to the HTTP API) to send the response data directly from the file.

There are lots of other reasons for using this mechanism – for example, Exchange 5.5’s POP, IMAP and NNTP servers used this technique to render RFC822 message content.  The Exchange server would render the email message to a temporary file, and then use TransmitFile to send the response to the client.  We used this for two reasons – we wanted to be able to respond to render failures above, but also because we needed to deal with potentially huge email messages.

Which is the other huge advantage of “temporary” temporary files over in-memory buffers - their size is bounded by the available disk space, NOT by available RAM.  So while an in-memory render might fail on a 1G file (because you couldn’t allocate a gigabyte of contiguous memory), it will work just fine to a “temporary” temporary file (assuming you have a gigabyte of disk space free on the rendering drive).  If you exceed available memory, the memory manager will flush the file data to disk. This causes a performance hit, but your operation will succeed instead of failing.

Comments

  • Anonymous
    April 19, 2004
    Hmm. In Unix-based systems such as OS X, this is done by creating a file in /tmp, opening the file (usually create and open are combined into one atomic step) and then deleting the file. The file will remain in memory until it is closed.
  • Anonymous
    April 19, 2004
    So the *nix filesystems and cache manager treat the /tmp portion of the namespace specially?

    Or is because the inode's been unlinked that makes it behave specially? What happens if the system comes under memory pressure and the file metadata needs to be flushed to disk?

  • Anonymous
    April 19, 2004
    what chris is talking about is different.

    first, /tmp isn't special.

    if you unlink an open file, the file isn't deleted from disk until the file is closed. basically all it does is remove the file from the directory's file, so it only exists in the inode allocation tables.

    it doesn't cause the file to live in ram, etc.

    basically there's no need to clean up your temp files.
  • Anonymous
    April 19, 2004
    Thanks Byron, I appreciate it.

    So the unlink does the equivilant of the FILE_FLAG_DELETE_ON_CLOSE, but not the combination of the two (which is what gets you the "never flushed to disk unless necessary")?
  • Anonymous
    April 19, 2004
    yup, that's correct.
  • Anonymous
    April 20, 2004
    The comment has been removed
  • Anonymous
    April 21, 2004
    As usual, insiteful comment, Mike.
  • Anonymous
    April 21, 2004
    Hm, "inciteful" or "insightful" ;)