EWS, MIME, and the missing Internet message headers

MIME content conversion in Exchange 2007 and Exchange 2010

Exchange 2007 and 2010 do not store MIME content. In general, MIME content is converted to MAPI properties for storage. When MIME content is requested from Exchange, the MIME content is formed from the MAPI properties.

Let’s start with the easy one, Exchange 2010. In most cases, Exchange 2010 MIME content that is returned by EWS contains all the expected header information. This is because content that is not converted to MAPI is stored in the PidTagMimeSkeleton (proptag 0x64F00102) property, as specified in [MS-OXCMSG]: Message and Attachment Object Protocol Specification section 2.2.1.28. As long as nothing modifies the item by using a remote operation (ROP) (think Outlook and MAPI), the Exchange 2010 MIME writer merges the MIME skeleton with content converted from MAPI, as specified in [MS-OXCMAIL]: RFC2822 and MIME to E-Mail Object Conversion Algorithm section 2.4. This case applies to pure MIME messages only. The original MIME content is never copied when it includes a TNEF body part. This is a concern if the originating client is an Outlook client that formats the body as Rich Text Format (RTF). If you are only targeting Exchange 2010, you can (under the conditions I described) expect Internet message headers in the MIME content. Still, you should be aware that if you are only requesting the MIME headers, the InternetHeaders collection is not complete. The InternetHeaders collection currently does not include all address headers, such as From and To.

Exchange 2007 does not include the PidTagMimeSkeleton property. The Exchange 2007 MIME reader converts the MIME content to MAPI and then discards the MIME content. The Exchange 2007 MIME writer always reconstructs the MIME content from the stored MAPI properties. 

Why is this? Exchange 2007 and Exchange 2010 implement MIME to MAPI to MIME conversion differently. The PidTagMimeSkeleton property introduced in Exchange 2010 maintains the header collection for an email object. For more information about MIME to email object conversion, see [MS-OXCMAIL].

Getting Internet message headers

To get the message headers, the EWS schema exposes the InternetMessageHeaders first-class property. Unfortunately, this property does not appear to return a complete set of message headers. For example, in Exchange 2010 Service Pack 1 (SP1), the InternetMessageHeaders property doesn't return address headers such as From and To. I suggest that you use the PR_TRANSPORT_MESSAGE_HEADERS property, as specified in [MS-OXPROPS]: Exchange Server Protocols Master Property List section 2.1103. You can use the EWS extended property feature to access PR_TRANSPORT_MESSAGE_HEADERS via EWS. For your convenience, the following is the property definition that you can add to your EWS Managed API PropertySet object to get the collection of headers.

ExtendedPropertyDefinition transportMsgHdr = new ExtendedPropertyDefinition(0x007D, MapiPropertyType.String);

OK, now that you have the PR_TRANSPORT_MESSAGE_HEADERS string, what are you going to do with it? You can parse this content in several ways. If your application runs on an Exchange server, we suggest that you use the MimeDocument class (available in Exchange 2007 and later versions). Otherwise, you can use CDOSYS (available in Windows 2000 and later versions). (For more information about CDOSYS, see Microsoft Support.) And of course, you can also write your own parser or use another publicly available parser.

So, you have the following options for using EWS to get specific MIME headers:

• Call the GetItem operation and request the ItemSchema.MimeContent property. This is the most expensive option. This option requires MIME conversion at the server, and after the conversion, you still have to decode and parse the headers.

• Call GetItem and request the ItemSchema.InternetHeaders property. This is a moderately expensive call. Although you don't have to retrieve the MIME content and therefore you don’t have the associated conversion costs, you still have to parse the headers. Also note that not all headers are returned in the InternetMessageHeaders collection, as I mentioned earlier.

• Call the FindItem or GetItem operations and request an extended property in the Internet headers namespace. This is the least costly option for getting specific headers. For information about how to do this, see Henning Krause's blog post Custom Mail headers and EWS.

The following example shows how to get specific MIME headers.

private static void MimeHeaderSample(ExchangeService service)
{
   const string mimeContentFormat =
         "Mime-Version: 1.0\r\n"
      + "From: michael@contoso.com\r\n"
      + "To: amy@contoso.com\r\n"
      + "Subject: nothing\r\n"
      + "Date: Tue, 15 Feb 2011 22:06:21 -0000\r\n"
      + "Message-ID: <{0}>\r\n"
      + "X-Experimental: some value\r\n"
      + "\r\n"
      + "I have nothing further to say.\r\n";
 
   // Upload the MIME content to the server.
   // This is similar to delivering it via SMTP.
   EmailMessage newMessage = new EmailMessage(service);
   string messageId = "" + Guid.NewGuid().ToString() + "@contoso.com";
   string mimeString = String.Format(mimeContentFormat, messageId);
   newMessage.MimeContent = new MimeContent("us-ascii", Encoding.ASCII.GetBytes(mimeString));
   newMessage.Save(WellKnownFolderName.Drafts);
 
   // The MIME content contains a header named "X-Experimental".
   // Create an extended property definition for retrieving the value of the X-Experimental header.
   ExtendedPropertyDefinition headerProperty = new ExtendedPropertyDefinition(
         DefaultExtendedPropertySet.InternetHeaders,
        "X-Experimental",
         MapiPropertyType.String);
   PropertySet columns = new PropertySet(
         BasePropertySet.IdOnly,
         EmailMessageSchema.InternetMessageId,
         headerProperty);
 
   // Show the message ID and X-Experimental header value
   // from the five newest messages in the Drafts folder.
   ItemView view = new ItemView(5 /* pagesize */);
   int i = 1;
   view.PropertySet = columns;
   view.OrderBy.Add(EmailMessageSchema.DateTimeReceived, SortDirection.Descending);
   Folder drafts = Folder.Bind(service, WellKnownFolderName.Drafts);
   FindItemsResults<Item> messages = drafts.FindItems(view);
   foreach (Item item in messages)
   {
      EmailMessage message = item as EmailMessage;
      if (message != null)
      {
         Console.WriteLine("{0}. {1}", i, message.InternetMessageId);
         foreach (ExtendedProperty property in message.ExtendedProperties)
         {
            if (property.PropertyDefinition == headerProperty)
            {
               Console.WriteLine("   X-Experimental header value: {0}", property.Value);
            }
         }
      }
 
      ++i;
   }
}

You can use the EmailMessage.Bind method (which uses the GetItem operation) to retrieve standard message properties and MIME header values from email messages. Latencies for GetItem requests can be larger than the latencies for FindItem requests. It is important to note that the InternetMessageHeaders collection as exposed as a first-class schematized property is, for the most part, only available by using the GetItem operation. I say for the most part because you can use the FindItem operation and request the header collection by using extended properties. Note that the FindItem operation only returns the first 512 bytes (255 Unicode characters) of any property; therefore, message header collections that are longer than 512 bytes will be truncated. The previous code example shows a suggested alternative that obtains specific MIME header values from messages without using the GetItem operation.

One thing to note about this code example is that, for Exchange 2010, the first time this code is run on a new mailbox, the value of a new custom MIME header will not be written to the stored message. This is because the x-header content is converted and because x-headers, as converted to named properties, must already be mapped for the specific user's mailbox. The mapping occurs upon the first request to add the named properties. Subsequent requests to create the named property will result in the property and value being stored on the message. The behavior for Exchange 2007 is a bit different. The first time an x-header is written to a mailbox database (compared to a mailbox on Exchange 2010), a mapping is created for the x-header to a named property across the mailbox database. Subsequent requests to create the named property will result in the x-header being processed and stored.

If the performance gain from this approach to retrieving MIME header values is worth the cost of changing your application, mapping the MIME header property should be made part of the provisioning process for new mailboxes, so that the properties are consistently available for all messages across all mailboxes. For background information about how the process works, see Named Properties, X-Headers, and You on the Exchange Team Blog.

Conclusion

Now I’ve told you how to work around the problem with MIME and the missing headers. But I have a better suggestion: Don’t use MIME! Your system administrators and your fellow mailbox users will appreciate it. MIME conversion is an expensive process. If you are using EWS, just use the first-class properties together with the extended properties to get the property set you want. Besides the lower conversion expense, you also have a finer degree of control over what properties you want returned. But if you really must use MIME, this article should help you avoid some confusion around message headers and MIME between Exchange 2007 and Exchange 2010.