DIME

Sending Files, Attachments, and SOAP Messages Via Direct Internet Message Encapsulation

Jeannine Hall Gailey

This article assumes you're familiar with SOAP, XML, and WSDL

Level of Difficulty123

SUMMARY

Direct Internet Message Encapsulation (DIME) is a new specification for sending and receiving SOAP messages along with additional attachments, like binary files, XML fragments, and even other SOAP messages, using standard transport protocols like HTTP. In this article, the author explains what DIME is and how it differs from MIME encapsulation. A detailed description of the message format and how it is parsed, as well as working with SOAP and extending it with WSDL, is also included.

Contents

DIME Basics
A DIME Example
Parsing a DIME Message
How is DIME Different from MIME?
Using DIME in a SOAP World
WSDL Extensions for DIME
DIME and the WSDK
Conclusion

If you've been keeping up on the latest innovations in the rapidly changing world of XML Web Services, you may have stumbled across a new specification called DIME (Direct Internet Message Encapsulation). As you might have guessed from its name, DIME is similar to MIME for Web Services, but there are some key differences between the two specifications. This article will expand upon the current draft specifications for DIME in an attempt to demystify this technology and show you how DIME can be used. Along the way, I'll discuss why it was necessary to add another standard to the rapidly crowding world of Web Services specs, and review how DIME relates to other technologies like SOAP and Web Services Description Language (WSDL).

DIME Basics

DIME is based on a specification recently submitted to the Internet Engineering Task Force (IETF), which details a method for sending and receiving SOAP messages along with additional attachments, like binary files, XML fragments, and even other SOAP messages. Like SOAP, DIME messages are sent using standard transport protocols, such as HTTP and TCP. For example, suppose that you are designing an XML Web Service that sends large media files to a requesting client. With currently available Internet protocols, you could certainly rig such a service. You might encode the binary attachments as base64 XML and include them in the body of the SOAP message. But what happens if the attachments are very large or digitally signed? What if you want to send XML fragments within a SOAP message that has a different character encoding? These are just a few of the instances when a solution based on DIME can greatly simplify your life.

As you know, SOAP provides a flexible and extensible way to send highly structured and typed XML data across the wire. Now that applications are able to communicate with each other in a platform-agnostic fashion using XML, the next question for application designers is how to use SOAP to send other important non-XML stuff, like binaries. The trick, of course, to sending binary data in a SOAP message is to use base64 encoding. Naturally, this requires an encoding or decoding step at each end of the process. Now you can begin to see how it's possible to run into some processor performance-related issues when you start sending really large binary files, such as JPEG image files or even digital sound and video files. Including binary files in SOAP messages using standard base64 encoding can also be tricky when the binary files are digitally signed. Even attaching an XML document or fragment to a SOAP message can pose problems, especially if the documents do not use the same character encoding as the primary SOAP message.

DIME allows you to send attachments of various types along with your SOAP message, even when the attachments in question do not fit conveniently or efficiently into an XML format. DIME is designed to be a fast and efficient protocol to parse. The length and type of attached data are defined in a few simple header fields. The protocol is kept lean and mean by the assumption that any additional message metadata will be included as part of a SOAP message, since SOAP is already such a rich metadata-based protocol. Although designed to work with SOAP, the use of DIME is not strictly limited to SOAP, and it may prove useful whenever a simple, efficient message encapsulation is required.

Figure 1** DIME Fields **

DIME specifies a way to include attachments with SOAP messages where the attachments can include binary image files, media files, or even other SOAP messages or XML fragments. DIME messages are created by a DIME generator and consumed by a DIME parser. A DIME message consists of a series of one or more DIME records. Each record is self-describing—that is, the record header contains binary information used by a parser to interpret the message. Figure 1 shows the layout of fields in a DIME record. Note that variable-length fields must be padded so that the length of the field is a multiple of a 4-byte interval. The fields shown in Figure 1 are defined in Figure 2.

Figure 2 DIME Message Data

A DIME Example

In order to better understand how DIME messages are used, let's see what a DIME message really looks like. Consider a sample XML Web Service that stores MPEG media files that are sent as DIME messages. Figure 3 shows what the DIME request message for this service might look like.

Figure 3 DIME Request Message

00001 1 0 0 0010 00000000000000000000 0000000000000000 0000000000101000 00000000000000000000000110110101 https://schemas.xmlsoap.org/soap/envelope <soap-env:Envelope xmlns:soap-env="https://schemas.xmlsoap.org/soap/envelope/" xmlns:msg="https://example.com/DimeExample/Messages/" xmlns:ref= "https://schemas.xmlsoap.org/ws/2002/04/reference/" > <soap-env:Body> <msg:GetMediaFile> <msg:fileName>myMediaFile.mpg </msg:fileName> <msg:file ref:location= "uuid:F2DA3C9C-74D3-4A46-B925-B150D62D9483" /> </msg:GetMediaFile> </soap-env:Body> </soap-env:Envelope> ------------------------------------------------------------------------- 00001 0 0 1 0001 00000000000000000000 0000000000101001 0000000000001010 00000000000101011010101011100000 uuid:F2DA3C9C-74D3-4A46-B925-B150D62D9483 video/mpeg <<First 1.42 MB of binary data for myMediaFile.mpg>> ------------------------------------------------------------------------- 00001 0 1 0 0000 00000000000000000000 0000000000000000 0000000000000000 00000000000010000110110001000000 <<Remaining 552 KB of binary data for myMediaFile.mpg>>

In this sample, the fixed-length fields of the message header are shown as their bit representations and the other fields are shown in the appropriate format. For readability, the records in the message are separated with horizontal lines and spacing has been added between fields. Also, to shorten the sample, the binary data has been removed.

This DIME message contains three records. The first is a SOAP message that references the attachment. The remainder of the message contains an MPEG video file attachment that has been chunked into two separate records.

Parsing a DIME Message

Now let's see how the DIME message in Figure 3 would be interpreted by a DIME parser. When the message reaches its destination, a DIME parser is invoked to read the message and extract its attachments. The parser moves through the DIME message in a serial fashion, reading the contents of each record one at a time. For the bit representation in Figure 1, the parser reads the bits in a big-endian manner in which the left-most bit is the most significant bit in the field. Starting with the first record in my sample message, the parser first checks the VERSION field to make sure that it can parse the message. If the parser is compliant with this version of DIME, it begins to read the rest of the header in the first record. If the parser is not compliant with or cannot recognize this version, then it should reject the message. All records in a message must have the same version number.

The parser then reads the message begin (MB) flag, which should be set in only the first record of the DIME message. If this flag is not set in the first record or is set in any subsequent records, then the entire message is considered malformed. The parser finds the MB flag set in the first record of my sample and continues on to the message end (ME) flag. As you might expect, if there are records after a record where the ME flag is set or if the ME flag is set in more than one record, the parser should reject the message as malformed. If the parser encounters both an MB and ME flag set on the same record, it knows that the message is only one record long. In my sample message, the ME flag is set only in the final record.

After the MB and ME flags, the parser reads the Chunk flag (CF), which is used to indicate a chunked payload. Chunking is a way to break a single attachment into multiple records, which is very useful when dealing with very large attachments. By breaking an attachment into chunks, you do not have to buffer the entire attachment and write it to a single DIME record. Instead, a large data stream can be written from a much smaller buffer into any number of fixed-length DIME records until the end of the stream is reached. For chunked records, the CF flag is set for the first and all subsequent chunked records, except for the final record chunk. In my example, I have broken the attached 1.97MB MPEG file into two chunks, with the first 1.42MB chunk in the second record and the remaining 552KB in the last record.

When reading a series of chunked records, the parser assumes that the first record without the CF flag is the final record in the chunk; in this case, it's the last record in my sample. The record type is specified only in the first record chunk, and all remaining chunks must have the TYPE_T field and all remaining header fields (except for the DATA_LENGTH field) set to zero. Notice that in the third record of my sample the only header fields that have a nonzero value are VERSION, ME, and DATA_LENGTH.

The parser then reads the TYPE_T field and uses the information to determine how to read the TYPE field when it gets to it. Currently, there are two main ways to specify the record TYPE: by absolute URI (0x02) and by MIME media-type (0x01). In my sample, the first record has a TYPE_T value of 0x02, indicating a URI-specified type, and the second record has a value of 0x01, indicating a MIME media-type-specified type. After TYPE_T, the RESERVED field is ignored by the parser since it is designated only for future use.

The parser then moves on to the four length fields in the header going in the following order: OPTIONS_LENGTH, ID_LENGTH, TYPE_LENGTH, and DATA_LENGTH. Each of these fields specifies the byte length of a corresponding data field in the message. Since the first three fields are unsigned 16-bit integers, each of their corresponding data fields can contain up to 64KB of data. Since DATA_LENGTH is an unsigned 32-bit integer, the DATA field can contain up to 4GB of data. Although this is a physical limitation on the amount of data in a single DIME record, there is no limit to the number of records in a DIME message. Since large attachments can be chunked, the DIME specification places no actual limit on the size of attachments.

After reading these fields, the parser moves on to the OPTIONS field. This data field, like the remaining ones, requires padding to the nearest 4-byte multiple. This means that if the OPTIONS_LENGTH value is 0x135, which sets the OPTIONS field length to 309 bytes, then the OPTIONS field must have an additional 3 bytes of padding. After the parser reads the specified 309 bytes, it knows to skip to the 313th byte and begin reading the next field. The OPTIONS field provides some measure of extensibility for DIME messages by allowing up to 64KB of additional data. However, new OPTIONS elements must be approved by a standards body before being added to the DIME specification. Each optional element contains the following three fields:

• ELEMENT_T is an unsigned 16-bit integer that specifies the encoding of the element.
• ELEMENT_LENGTH is an unsigned 16-bit integer that specifies the length of the element, up to 64KB.
• ELEMENT_DATA contains the data for the element. This field can be up to 64KB in size and must be padded to the nearest 4-byte multiple.

If a parser cannot recognize an optional element, then it simply ignores it and continues processing the message.

After processing the optional elements, the parser reads the ID and TYPE fields. The ID field specifies a URI that uniquely identifies the record. In my sample, the SOAP message in the first record is able to reference the attachment using the ID value in the second record. The TYPE field specifies the type (and thereby the encoding) of the data in the record. Depending on the value of the TYPE_T field, the type is referenced either by URI or by MIME media-type. In my sample, the encoding for the SOAP 1.1 message is referenced by the URI https://schemas.xmlsoap.org/soap/envelope, but I could have also referenced the media-type "application/soap+xml" if I had set TYPE_T to 0x01. The attachment was referenced as "video/mpeg" with the TYPE_T field set to 0x01 to use the MIME media-type reference.

Finally, there is the DATA field, which contains the payload of the record. In my example, the parser reassembles the chunked MPEG attachment by reading the data from the second and third records and serializing the complete data file.

To get your hands on a real DIME parser (without writing one yourself) and some complete service samples, download the SOAP Toolkit 3.0.

How is DIME Different from MIME?

One of the most common questions asked about the DIME specification is why we need this new encapsulation standard when we already have MIME, a widely used protocol developed to enable the transmission of attachments with e-mail via mail transport protocols like SMTP. Like DIME, the MIME multipart protocol (RFC 2387) also allows you to encapsulate multiple attachments of different media types in a single message. MIME depends on the use of special strings to separate multiple message attachments and allows you to include additional metadata in the message by the creation of custom message headers. While these features make MIME very flexible and explain its wide use in today's Internet, they also tend to make it a less efficient protocol.

While potentially less flexible than MIME, DIME prescribes a simpler message format, which in turn provides for faster and more efficient message processing. For example, all of the data in a MIME message must be read and interpreted to determine simple things like the number of attachments included in the message. However, when using DIME, a parser can simply use the data in the record headers to quickly walk through and count the number of records in the message without having to read any record data. Also, in a DIME message, the only metadata included in the record headers specifies the length and encoding of the message header fields and payload. Any additional message information can be included in the attached SOAP messages. For a more detailed discussion of the benefits of DIME over MIME for Web Services, see the excellent MSDN® article "DIME: Sending Binary Data with Your SOAP Messages," by Matt Powell.

That said, the purpose of DIME is not to replace MIME, but rather to build a new specification that incorporates the best features of MIME which work well in the new Web Services world. In fact, so many MIME media-type definitions already exist that DIME uses these existing media types to identify the encoding of data in DIME records. That is not to say that in the future you won't be sending DIME attachments via SMTP.

Using DIME in a SOAP World

Although there is no requirement that a DIME message contain a SOAP message or vice versa, SOAP is the primary reason for the creation of DIME. When using DIME to encapsulate SOAP messages, there are some additional rules to follow, which can be found in the WS-Attachments specification. A draft of this proposed specification can be found at https://www.ietf.org/internet-drafts/draft-nielsen-dime-soap-01.txt.

Here are some things to keep in mind when using DIME to send SOAP messages:

• The first DIME record contains the primary SOAP message. Additional SOAP attachments are included in subsequent record payloads.
• The primary SOAP message may cross-reference any subsequent attachments by a Universal Unique Identifier (UUID) or any form of URI. This UUID is indicated by the href attribute and is used to match the ID field of the corresponding DIME record.
• When cross-refer-encing attachments, relative URI references should be converted to absolute URI references.
• When binding DIME messages to HTTP, the HTTP Content-type header field must specify "application/dime" instead of the usual "application/soap+xml" or "text/xml" defined respectively by the SOAP 1.2 and SOAP 1.1 protocols.
• For DIME attachments that are signed or encrypted, security information about the attachment should be included in the header of the primary SOAP message.

WSDL Extensions for DIME

WSDL provides a method for describing how to access XML Web Services over the Internet, which includes specifying the protocol bindings for SOAP messages. Since DIME messages are different than SOAP messages, this necessitates an extension to WSDL for binding the transport layer (usually HTTP, but possibly TCP) to DIME rather than to SOAP. These WSDL extensions are discussed in the proposed standard WSDL Extension for SOAP in DIME. Although I will not discuss it in detail, this specification defines the additional XML elements and attributes in WSDL needed to describe DIME-consuming Web Services. Also, the Microsoft® SOAP Toolkit 3.0 contains a WSDL generator that can be used to create DIME-aware WSDL files for your DIME-based XML Web Service.

The major change to WSDL to support DIME involves the addition of the <dime:message> element in the <wsdl:input> or <wsdl:output> elements of a Web Services operation bindings definition. The <dime:message> element must include the wsdl:required attribute set to true along with a newly defined layout attribute. This new attribute is used to specify how attachments in a DIME message are referenced by the primary SOAP message. There are two possible URI values for the layout attribute. The URI https://schemas.xmlsoap.org/ws/2002/04/dime/closed-layout specifies that all parts of a DIME message be referenced from the primary SOAP message and in the proper order. Alternately, https://schemas.xmlsoap.org/ws/2002/04/dime/open-layout specifies that additional attachments can be included in a DIME message even though they are not referenced by the SOAP message, provided that they come after all of the attachments in the DIME message that are referenced by the SOAP message.

In addition to these new element definitions, WSDL extensions for DIME also allow the addition of the following three elements in a WSDL file to further describe the nature of DIME attachments used by the service: content:type specifies the value type of an attachment, content:mediaType specifies the MIME media-type of an attachment, and content:documentType specifies the value type of an XML document.

Figure 4 is a sample WDSL file that describes the DIME-based Web Service that I have been discussing.

Figure 4 WSDL for DIME Service

<?xml version="1.0" ?> <wsdl:definitions name="DIME Sample" targetNamespace="https://example.com/DimeExample" xmlns:wsdl="https://schemas.xmlsoap.org/wsdl/" xmlns:soap="https://schemas.xmlsoap.org/wsdl/soap/" xmlns:dime="https://schemas.xmlsoap.org/ws/2002/04/dime/wsdl/" xmlns:content= "https://schemas.xmlsoap.org/ws/2002/04/content-type/" xmlns:ref="https://schemas.xmlsoap.org/ws/2002/04/reference/" xmlns:svc="https://example.com/DimeExample/Service/" xmlns:msg="https://example.com/DimeExample/Messages/" xmlns:xs="https://www.w3.org/2001/XMLSchema" > <wsdl:types> <xs:schema targetNamespace="https://example.com/DimeExample" elementFormDefault="qualified"> <xs:import namespace= "https://schemas.xmlsoap.org/ws/2002/04/content-type/"/> <xs:import namespace= "https://schemas.xmlsoap.org/ws/2002/04/reference/"/> <xs:element name="GetMediaFile"> <xs:complexType> <xs:sequence> <xs:element name="fileName" type="xs:string"/> <xs:element name="file" type="msg:MediaFile" minOccurs="0"/> </xs:sequence> </xs:complexType> </xs:element> <xs:complexType name="ReferencedBinary"> <xs:simpleContent> <xs:extension base="xs:base64Binary"> <xs:attribute ref="ref:location" use="optional"/> </xs:extension> </xs:simpleContent> </xs:complexType> <xs:complexType name="MediaFile"> <xs:simpleContent> <xs:restriction base="msg:ReferencedBinary"> <xs:annotation> <xs:appinfo> <content:mediaType type="video/mpeg"/> </xs:appinfo> </xs:annotation> </xs:restriction> </xs:simpleContent> </xs:complexType> </xs:schema> </wsdl:types> <wsdl:message name="GetMediaFile"> <wsdl:part name="body" element="msg:GetMediaFile"/> </wsdl:message> <wsdl:portType name="PortType"> <wsdl:operation name="GetMediaFile"> <wsdl:input message="svc:GetMediaFile"/> </wsdl:operation> </wsdl:portType> <wsdl:binding name="SoapDimeBinding" type="svc:PortType"> <soap:binding transport="https://schemas.xmlsoap.org/soap/http" style="document"/> <wsdl:operation name="GetMediaFile"> <soap:operation soapAction="https://example.com/GetMediaFile"/> <wsdl:input> <dime:message layout= "https://schemas.xmlsoap.org/ws/2002/04/dime/closed-layout" wsdl:required="true"/> <soap:body parts="body" use="literal"/> </wsdl:input> </wsdl:operation> </wsdl:binding> <wsdl:service name="DimeSample"> <wsdl:port name="SoapDimePort" binding="svc:SoapDimeBinding"> <soap:address location="https://example.com/DimeSample/"/> </wsdl:port> </wsdl:service> </wsdl:definitions>

DIME and the WSDK

In order to drive the adoption of DIME and a number of other newly proposed Web Services standards, Microsoft previewed a product called the Web Services Development Kit (WSDK) in August 2002. This development kit is a .NET managed code assembly that implements a set of classes that can be used to develop XML Web Services in ASP.NET that support the WS-Security, WS-Routing, WS-Attachments, and DIME standards. These classes essentially implement filters in ASP.NET that intercept outbound Web Service requests and inbound Web Service responses and either serialize the required SOAP elements so that they conform to the specifications or decompose them into their corresponding objects. This means you can use Visual Studio® .NET to create a DIME-enabled Web Service using these managed classes. For more information, see the Web Services Enhancements.

Conclusion

DIME has the potential to become a very useful encapsulation method for attachments to SOAP messages by utilizing the rich metadata in SOAP against a simple, efficient encapsulation mechanism. In addition to its technical merits, DIME is receiving the full support of Microsoft going forward, as is indicated by its inclusion in the newest version of the Microsoft SOAP Toolkit. As with any early specification-based technology, you can expect some changes as the technology matures, so keep your eyes open for updates to the DIME and related specifications.

For related articles see:
WSDL Extension for SOAP in DIME
Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies

Jeannine Hall Gaileyis a freelance writer and Web consultant who specializes in Web Services technologies. She has written numerous articles and two forthcoming books on Web Services with Microsoft Press. Visit her Web site at https://www.webbish6.com.