Microsoft Surface 1.0 SP1: SDK - Tagged Objects
Overview
The Microsoft Surface Vision System captures and processes raw images of a Microsoft Surface screen and can "see" the actual outlines of physical objects that are placed on the screen by using cameras that operate in the near infrared light (specifically, 850 nm).
(For an example of how to see object outlines, see Capturing and Rendering a Raw Image.)
However, if you want your Microsoft Surface application to be driven by objects, the Microsoft Surface Vision System also recognizes special tagged objects that are marked with a special pattern of dots called tags. A tag consists of a geometric arrangement of infrared reflective and absorbing areas. Tagged objects give your application several advantages over processing raw images:
Tagged objects are more computation-efficient. The tags are small and well-defined, so specialized code in the Vision System can locate and track them quickly, accurately, and efficiently.
The Microsoft Surface platform inherently recognizes tags, so it is easy to write an application that uses tags. (In contrast, an application that uses raw images must include its own image-recognition algorithms.)
Each tag stores a distinct binary code value, so an application can distinguish one object from another.
Microsoft Surface supports two types of tags:
Byte tags store 8 bits of data (1 byte), so there are 256 unique tag values. The value of a byte tag is represented in code by a ByteTag structure.
Identity tags store 128 bits of data (two 64-bit values), so there is an effectively unlimited range of unique tag values. The value of an identity tag is represented in code by an IdentityTag structure.
Your application's needs determine which type of tag that you should use.
When you use tagged objects, you can use as many instances of the same tag as an application requires. However, if a user moves a tagged object too fast, the Microsoft Surface Vision System cannot track the object successfully. If you're writing a Microsoft Windows Presentation Foundation (WPF) application, you can create an on-screen visual object that tracks the tag motion by using the TagVisualizer control.
How to Use Tagged Objects
A Microsoft Surface application can use tagged objects in several ways:
Recognize objects: An application can use tags to recognize an object or distinguish among a collection of objects. For example, a board game might use different tags to represent the players and trace their positions on the Microsoft Surface screen. Similarly, a chess application might use eight instances of one tag to represent the white pawns, two instances of another tag for the two rooks, two instances of another tag for the knights, two instances of another tag for the bishops, one tag for the queen, and another tag for the king. Therefore, the application would use six unique tags for the white pieces, and another set of six unique tags for the black pieces.
Start a command: An application might use a tag to start a command or action. For example, placing a tagged object on the Microsoft Surface screen might open a menu or start a video.
Start or display an application: Applications can register a particular byte tag value or identity tag series. When a user places the relevant tag on the Microsoft Surface screen, it displays a menu that enables the user to start or display any application that has registered the tag. If a user selects an application by using a tagged object, the application designer can use that tag contact as the starting point for the user's experience with that application. This feature is called object routing.
Point and orient applications: An application might require precise movement with absolute orientation. For example, an application might enable a user to move in sequence through a series of video clips, turn the pages in a virtual book, or adjust the volume of a video or audio clip. To adjust the volume, the application might enable the user to turn a tagged object in a circular motion, like the dial on a radio or television. (Microsoft Surface applications enable users to move and rotate an object with their fingers. But tags provide a more precise absolute location than fingers, so applications can use tagged objects as pointing devices.)
Because identity tags have a greater range of possible values, they also enable additional scenarios:
Casino rewards promotions: A casino marketing department produces an offer for $100 in free slot play. The printed offer includes the identity tag with the customer ID and a unique offer code, such as a casino chip. The customer brings this offer to the venue and places it on a Microsoft Surface screen. The customer code and offer are validated, and a $100 credit is associated with the customer account as downloadable credits. The next time the customer inserts their loyalty card into a slot machine, they receive the $100 credit to the machine.
Theme park photo pass: Passes are issued to guests of the park each day. The pass has an identity tag ID on it. As photos are taken throughout their visit to the park, the digital pictures are associated with the ID on the photo pass. The guests then place the pass on a Microsoft Surface screen and all the photos that are associated with the pass appear on the screen for the guests to view, manipulate, and purchase.
Hotel loyalty cards: Identity tags are placed on loyalty cards to identify customers. Premier customers have access to more services and applications on a Microsoft Surface unit than typical customers or those without a loyalty card.
Fast food kids' toys: Identity tags are placed on kids' toys. A user places the toy on a Microsoft Surface screen and the application responds to the toy in some way.
Note
Any application that uses tagged objects should register the tags that it uses. For more information, see Registering a Microsoft Surface Application.
Security Considerations for Tagged Objects
Do not use tagged objects to access private data. Tags are not sufficiently secure to use as a method of authentication. You can use tagged objects to identify a user, but you should require the user to enter a password, personal identification number (PIN), or other secure data for authentication purposes.
When you use tags, consider the following important security issues:
An identity tag does not have enough bits to be cryptographically secure. Do not use an identity tag as a password equivalent.
Tag bits are not encrypted when they are in memory. For example, they are not equivalent to the System.Security.SecureString class in the .NET Framework. When you write code that works with tags, imagine that you are working with strings. That is, in your application, is System.String sufficiently secure, or would you need to use SecureString? If you would use SecureString, do not use an identity tag for the data.
Depending on how tags are printed, photocopiers, digital cameras, and other devices can easily capture the tags so they are easy to duplicate.
Overall, we recommend that you can use tags for identification but you should not use them authentication. An identity tag is simply a way to enter shorthand, intended to save you from having to type something into a text box. It's the equivalent of having a hyperlink that you can click, instead of forcing you to enter a URL, or the equivalent of pressing CTRL+R to respond to an e-mail, instead of entering the recipient's address manually. The contents of an identity tag are not more secure than a URL or an e-mail address.
Byte Tags
Byte tags are one type of tagged objects that Microsoft Surface recognizes. These tags store 8 bits of data (1 byte), so there are 256 unique tag values. (In contrast, identity tags store 128 bits of data.)
http://i.msdn.microsoft.com/dynimg/IC353789.jpg
The value of a byte tag is represented in code by a ByteTag structure. The Value property represents the value of the tag that your application can use to interpret the physical object. The Contact.Tag.Byte.Value property stores the value for a given Contact.
There are 256 possible byte tags, each with an 8-bit value and orientation markers.
Byte Tag Geometry
Each byte tag that the Microsoft Surface Vision System identifies has an orientation and an 8-bit tag value. The following figure illustrates the basic tag that all other tags are based on.
http://i.msdn.microsoft.com/dynimg/IC357349.jpg
Infrared-absorbing background
One infrared-reflecting circle (0.125-inch radius) in the center of the tag. This circle locates the tag on the Microsoft Surface screen.
Three infrared-reflecting circles (0.08-inch radius) located 0.28 inches from the center of the tag in each direction (left, right, and down). These "guide" circles determine the tag orientation.
In addition, each tag contains from zero to eight data bits that define the tag value. These data bits are white circles (0.075-inch radius) that are centered in one of the following locations:
0.3058 inches horizontally and 0.2038 inches vertically (bits 1, 2, 5, and 6) from the center of the tag.
0.2038 inches horizontally and 0.3058 inches vertically (bits 0, 3, 4, and 7) from the center of the tag.
Byte Tag Values
There are 256 possible unique byte tags that are encoded as a byte. The highest order bit (bit 7) is at the 1 o'clock position when you look at the printed side of the tag. Less significant bits are then read counter-clockwise from the 12 o'clock position. An infrared-reflective circle in a bit location represents 1. The absence of a circle represents 0 (zero).
http://i.msdn.microsoft.com/dynimg/IC357350.jpg
http://i.msdn.microsoft.com/dynimg/IC357351.jpg
Printing Byte Tags
Microsoft Surface units include two full sets of byte tags (256 tags in each set). You can also create and print your own tags, according to the specifications in this article, up to the maximum of 256 unique patterns.
If you want to create and print your own byte tags, you have some freedom when you design and work with the physical tags. The following illustration shows the dimensions of a physical tag.
http://i.msdn.microsoft.com/dynimg/IC357355.jpg
The undefined region indicates where you can extend the geometry of the tag, if you need to.
The tag region is the opaque black-and-white patterned area of the actual tag. You can remove or cut a 0.125 × 0.125 inch triangle from each corner of the 0.75 × 0.75 inch square tag without affecting performance, if you need to affix the tag to a smaller circular object. However, the entire tag region must touch the Microsoft Surface screen.
Identity Tags
Identity tags are one type of tagged objects that Microsoft Surface recognizes. These tags store 128 bits of data (two 64-bit values), so there is an effectively unlimited range of unique tag values. (In contrast, byte tags store 8 bits [1 byte] of data.)
http://i.msdn.microsoft.com/dynimg/IC353790.jpg
The value of an identity tag is represented in code by an IdentityTag structure. The IdentityTag structure consists of two properties, Series and Value, which are 64-bit numbers.
An identity tag can represent a much greater range of values that an byte tag. This greater range of possible values enables the following possible scenarios:
Casino rewards promotions: A casino marketing department produces an offer for $100 in free slot play. The printed offer includes the identity tag with the customer ID and a unique offer code, such as a casino chip. The customer brings this offer to the venue and places it on a Microsoft Surface screen. The customer code and offer are validated, and a $100 credit is associated with the customer account as downloadable credits. The next time the customer inserts their loyalty card into a slot machine, they receive the $100 credit to the machine.
Theme park photo pass: Passes are issued to guests of the park each day. The pass has an identity tag ID on it. As photos are taken throughout their visit to the park, the digital pictures are associated with the ID on the photo pass. The guests then place the pass on a Microsoft Surface screen and all the photos that are associated with the pass appear on the screen for the guests to view, manipulate, and purchase.
Hotel loyalty cards: Identity tags are placed on loyalty cards to identify customers. Premier customers have access to more services and applications on a Microsoft Surface unit than typical customers or those without a loyalty card.
Fast food kids' toys: Identity tags are placed on kids' toys. A user places the toy on a Microsoft Surface screen and the application responds to the toy in some way.
Creating Identity Tags
The Printing Tagged Objects section describes general requirements for creating and printing tagged objects.
If you want to create and print your own identity tags, you can use the Identity Tag Printing Tool (GenTagUI.exe) to create identity tags by using a regular printer. You can also create identity tags by using the IdentityTagGenerator class in your own application.
After you create an identity tag, attach it to something solid, like a plastic card or heavy cardboard, because the tag must be perfectly flat for the Microsoft Surface Vision System to detect it. The black ink that is printed on an identity tag must be as dark as possible for the correct contrast between the white dots and the black background.
The identity tag printing tool enables you to generate a new series (Series) and values (Value) within that series. For example, consider a scenario like a promotion where Series represents a code for the promotion itself and Value represents individual offerings or even individual customers that the promotion targets.
After you create the Series and Value value, you can copy and paste them into your database or application code in XAML or C# format. You can also print the actual tag graphic to a black-and-white printer.
Designing Applications for Tagged Objects
Tagged objects provide many new scenarios for Microsoft Surface applications. However, when you create an application that is driven by tagged objects, there are unique design considerations to remember.
For an example of an application that is designed for tagged objects, see the Item Compare sample application. For a tutorial about how Item Compare is designed as a tag-driven application, see Designing Tag-Driven Applications for Microsoft Surface.
Choosing a Tag Type
Microsoft Surface supports two types of tags: byte tags and identity tags. Your application's needs determine which type of tag that you should use.
Byte tags store 8 bits of data (1 byte), so there are 256 unique tag values. Byte tags measure ¾ inch on each side, and the Vision System can reliably track them at relatively high speeds (over 4 inches per second) as they move across a Microsoft Surface screen.
Identity tags store 128 bits of data (two 64-bit values), so there is an effectively unlimited range of unique tag values. Identity tags measure 1 inch on each side, and they function best when they are stationary or nearly stationary. The Vision System can track identity tags at a maximum speed that is considerably lower than the maximum speed for byte tags.
Because byte tags are more compact and are faster for the Vision System to recognize, they represent the best choice for an object-driven application if the number of unique tag values that you need is relatively small (for example, identifying pieces in a board game). However, byte tags have only 256 possible values, so they may not be appropriate for some applications.
If your application must distinguish among a large number of unique values (for example, if the tag represent a unique customer ID from a large database), identity tags are the best choice.
Note
Although you can use identity tags to provide logon credentials, you should not use them as the only means of authentication because malicious users can copy or guess tags. Any authentication scheme that is based on tag values should also require another form of information, such as a PIN or password.
Reacting to Tagged Objects
If you are creating an application based on the Presentation layer, your application can respond to tagged objects most easily by using the TagVisualizer control. This control can recognize, track, and show user interface (UI) elements for tags that are placed on the control.
If you want to use custom processing (or if you are writing an application based on the Core layer), your application can respond to tagged objects by watching for Contact objects whose IsTagRecognized property is set to true. When that property is true, you can use the Tag property on the contact to get information about the type and value of tag that has been recognized.
Starting Applications with Tagged Objects
A user's first experience with your application is how to start it. If your application is designed to be driven primarily by tagged objects, consider integrating it with the object routing feature. Object routing enables users to start your application without going through Launcher. Ideally, after a user opens your application by using a tagged object, the application will use the tag value in some way.
About Printing
A Microsoft Surface application relies solely on the orientation and position of a tag (instead of the physical object that the tag is attached to), so the Microsoft Surface platform needs to know where the tag is located on the physical object the tag is placed. The location enables the platform to infer the location of the object based on the location of the tag. Depending on the physical requirements of your tagged object, you might need to place the tag somewhere other than its center. In such a case, you should correct for the location of the tag so your application can show the UI for the tag in the appropriate position.
If you use the TagVisualizer control, you can use the PhysicalCenterOffsetFromTag and OrientationOffsetFromTag properties of the TagVisualization object to correct for tag placement.
For more information about how to create and print tags for the physical objects, see Printing Tagged Objects below. For information about how to create and print identity tags, see Identity Tag Printing Tool.
Printing Tagged Objects
Microsoft Surface units include two full sets of byte tags (256 tags in each set). You can also create and print your own byte tags and identity tags.
In addition to the information in this topic, review the specifications for each tag type in the Byte Tags and Identity Tags sections above. If you are printing identity tags, you should also use the identity tag printing tool.
Reflectance
Make sure that printed tags meet the following requirements:
The white dots must reflect at least 73% of 850 nanometer (nm) infrared light but not more than 86%. Ultra high reflectivity whites can cause "bloom," which impairs the vision system's ability to recognize the tag.
The dark region must not reflect any more than 7% of 850 nm infrared light (93% or more should be absorbed).
Note
Limited tests show that a non-IR reflective or black background that is at least 1 inch larger than the tag itself can improve how tags are recognized, even if they are on a background with reflectance higher than 83%.
A single tag on a physical object with a black or non-IR reflective background provides the best tag recognition performance.
You can measure reflectance for surfaces with diffuse reflectance by using a reflectometer that has response in the near IR wavelength range (for example, the InspectIR-Vis instrument from Surface Optics Corp).
Materials and Placement
Whenever possible, place tags on infrared-absorbent objects. A tag on an object that is infrared-absorbent typically performs better than a tag on a highly infrared-reflective item. If you place a tag on a highly infrared-reflective surface, the Microsoft Surface Vision System might have trouble recognizing the tag.
Note
The infrared reflectivity of an object is not necessarily the same as its brightness to the naked eye. For example, some materials that appear dark to the eye can actually be reflective to infrared light.
You must place tags on a solid, consistent background (for example, with no patterns or stripes). Use a dark-colored or non-IR reflective background instead of a light background.
Place tags on flat surfaces that help the tag maintain flush contact with the tabletop. Also, tags work better on physical objects that do not easily tip over while they move.
The following materials and processes have been validated and shown to produce tags that function correctly with Microsoft Surface. If you use these materials, your tags should work properly. However, always test new tag types in their final format before you distribute them.
High IR-Reflective/White
White office paper
White Mylar
White PVC
White Styrene
White print labels (for example, Brother TZ-261)
IR-reflective dielectric coating on PET film
In general, most white paper works well. However, some are too bright and can cause a blur in the Microsoft Surface Vision System.
Low IR-Reflectance/Black
Laser printer black ink
Pantone Black C
Epolight 8771 IR absorbing ink
Top coats and protective layers
In practice, top coats such as UV print varnish and laminates can successfully improve a tag's durability and minimally affect the optical properties of the tag. However, many types of protective layers can significantly affect optical performance. You should test any protective layer that you add to a tag to make sure it performs correctly.
Printers
Ink jet printers that use a mix of cyan, yellow, and magenta to create black generally do not work. The three-color combination is typically invisible in the infrared spectrum. You should test all printing methods for their appearance in the infrared 850 nm wavelength.
Be careful that the tags cannot be rubbed off. Tags that are printed by using ink jet or laser printers wear off rapidly when they are used. If the tag printing wears off, you cannot use the tag anymore and the tag will smudge the Microsoft Surface screen. The tags that are included with the unit are printed on self-adhesive vinyl to protect them from wearing off.
Note
Do not use the Microsoft XPS Document Writer to create an .xps file for later printing of the identity tag. The XPS Document Writer introduces light gray lines that outline the small hexagonal areas of the tag. These lines can prevent the Vision System from detecting the tag.
For more information about how to print identity tags, see Identity Tag Printing Tool.
Using Object Routing with Tagged Objects
Typically, users open Microsoft Surface applications through Launcher. However, Microsoft Surface also includes object routing, which enables you to configure tagged objects that open your applications without using Launcher.
Tagged Objects and Applications
When a tagged object is placed down on a Microsoft Surface screen, the Microsoft Surface software reads the object's tag value and checks if the tag is registered with Surface Shell.
Note
Even if your application does not use object routing, you should register tags that your application uses to prevent interference from applications that do use object routing.
The Microsoft Surface software also checks whether any tag is registered for object routing. If a tag is registered for object routing, the Microsoft Surface software displays a menu that includes all of the applications that are registered for object routing with that tag.
When you design an application that works with object routing, we strongly recommend that your application use the tag value for some clear purpose after users open the application from the object routing menu. Strong designs provide immediate, obvious information to the user based on that tag. Users should instantly see what options are available and how to proceed based on the tag that they placed on the screen. For example, the tag value could correspond to a particular promotion or set of user preferences.
You can associate tagged objects with your application by using the application's XML registration file. For more information, see Registering a Microsoft Surface Application.
Object Routing UI
If a user places a tagged object that is registered for object routing on a Microsoft Surface screen, the Microsoft Surface software displays the object routing UI, as the following illustration shows.
http://i.msdn.microsoft.com/dynimg/IC353849.jpg
The object routing UI displays a menu of applications that users can open because they are registered with the tagged object. If an application does not have an icon image, the object routing UI displays a default icon.
The object routing UI displays on top of all other elements in the experience except the access points, but it does not prevent users from interacting with the current application. However, the current application does not interact with the tagged object that is displaying the object routing UI.
The Microsoft Surface software automatically sizes the object routing UI based on the tagged object's bounding box. For more information about the bounding box, see the Specifying Physical Attributes of Tagged Objects section below.
Note
We recommend that you register up to four applications for object routing with each tag.
If the object routing UI for two tags appears too close to each other on a Microsoft Surface screen, they repel each other. (This behavior avoids confusion that could occur if the UI elements overlapped.)
Unlike other UI elements, users cannot push an object routing menu beyond the edge of the screen, so all application icons in the menu are always visible.
When a user starts an application by using the object routing UI, the orientation of the tagged object determines the orientation for the application.
The object routing UI disappears when a user opens an application that is associated with the tag. The tagged object then interacts with the newly opened application. The object routing UI also disappears if the tagged object is removed from the unit.
Setting Up Object Routing
To configure object routing for an application, you must do the following:
Enable object routing on the Microsoft Surface Unit. For more information, see Configuring Object Routing on a Microsoft Surface Unit.
For each tagged object's value that you want to configure for object routing, define the tagged object in the registry of the unit.
For each application that is associated with a tagged object, register the application with the associated tagged objects in the application's XML registration file. For more information, see Registering a Microsoft Surface Application.
Specifying Physical Attributes of Tagged Objects
When a user places a tagged object on a Microsoft Surface screen to interact with applications by using object routing, the Microsoft Surface software displays a shadow for the object, a text label, and icons to start or display applications.
For each tagged object's value that you want to configure for object routing, you must define the tagged object in the registry of the Microsoft Surface unit. The values must be in hexadecimal format. Maintain the values for each defined byte tag value and identity tag series in the following registry locations:
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\TagInfo\v1.0\ByteTags\<Byte tag value>
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\TagInfo\v1.0\IdentityTags\<Identity tag series>
The following table describes the registry settings that you must define to describe the values of the tagged objects.
| Key | Description |
|---|---|
BoundingBoxWidth |
The width of the bounding box, in inches. (For more information, see the following "Bounding Box" section of this topic.) |
BoundingBoxHeight |
The height of the bounding box, in inches. (For more information, see the following "Bounding Box" section of this topic.) |
PhysicalCenterOffsetFromTagX |
The offset of the bounding box center from the tag center, in the x-direction. (For more information, see the following "Horizontal and Vertical Offsets" section of this topic.) |
PhysicalCenterOffsetFromTagY |
The offset of the bounding box center from the tag center, in the y-direction. (For more information, see the following "Horizontal and Vertical Offsets" section of this topic.) |
OrientationOffsetFromTag |
The offset of the bounding box orientation from the tag orientation, in degrees. (For more information, see the following "Orientation Offset" section of this topic.) |
Text |
The text to appear under the object. |
The following illustration demonstrates the following parts of a tag on a tagged object:
The offset of the bounding box center from the tag center, in the x-direction (PhysicalCenterOffsetFromTagX)
The offset of the bounding box center from the tag center, in the Y-direction (PhysicalCenterOffsetFromTagY)
The center point of the bounding box
The center point of the tag
The offset of the bounding box orientation from the tag orientation, in degrees (OrientationOffsetFromTag)
http://i.msdn.microsoft.com/dynimg/IC357333.jpg
You can translate (localize) the text that appears under a tagged object and include a separate registry key for each locale that you support. For example, the following registry key includes text for France:
- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\TagInfo\v1.0\IdentityTags\<identity tag series>\fr-FR\Text
If you do not specify a localized version for a locale, the default text is displayed.
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Surface\TagInfo\v1.0\IdentityTags\<identity tag series>\Text
Note
Keys that require numeric values do not accept localized values. For example, always use a period (".") as the decimal separator.
For more information about how to create the physical tags, see Printing Tagged Objects.
Bounding Box
To determine the size of the object routing UI, the Microsoft Surface software requires the size of the rectangle around the object, called the bounding box. The center point of the bounding box serves as the anchor for the tethers that connect the menu items to the tagged object in the object routing UI.
The bounding box is represented with values, in inches, for the BoundingBoxWidth and BoundingBoxHeight registry values.
Horizontal and Vertical Offsets
You can place a tag anywhere on the relevant surface of a tagged object. It does not have to be centered within the bounding box. For the Microsoft Surface screen to accurately reflect the position of the object, you must calculate the offset from the center of the tag to the center of the tagged object's bounding box.
http://i.msdn.microsoft.com/dynimg/IC357334.jpg
Offsets are represented in inches from the center of the tag to the center of the X or Y axis of the bounding box.
Orientation Offset
You can also place a tag in any orientation with respect to a tagged object. For the Microsoft Surface screen to accurately reflect the position of the object, you must calculate the offset between the up orientation of the tag and the up orientation of the object's bounding box.
Orientation offsets are represented in degrees from the orientation of the tag.
Installation Considerations
Ideally, when you develop your application, you should also create an installer that installs the application, creates the correct entries in the ApplicationName.xml registration file, and creates and sets the appropriate registry keys.
The installer should also check the registry for registered tagged objects. If the value or series of a tag has been specified in the registry, enable the administrator to overwrite the registry values or cancel the installation.
Note
Because byte tags have fewer available values, the possibility of registry conflicts is greater. The possibility of identity tag collision when the tag values are generated randomly is close to zero.
Automatically Registered Applications
When you start your application from Microsoft Visual Studio or if start it by running the executable file in administrator mode, the application is automatically registered with Surface Shell. However, applications that you open in this way cannot use object routing unless you have previously registered them with Surface Shell. For more information about how to register an application, see Registering a Microsoft Surface Application.