Creating Windows Touch Control Frameworks in C++ (PhotoStrip)
At PDC 2009, Reed Townsend presented some very exciting multitouch samples in a presentation: Windows Touch Deep Dive. The following video shows the presentation:
Many customers have been asking about how to properly handle multiple window objects and the first half of his presentation does a great job explaining the best practices. It's been a while since Reed presented this sample but I have the source code and did a short write up interpreting how everything is working. You can download the Windows Touch Photostrip sample from my server and the following notes explain the various pieces.
Summary
The PhotoStrip control created for this demo is similar to the one that is included in the Touch Pack and on the Microsoft surface. For this demo, not only is the control created, but an entire framework for having multiple reusable controls that can be mixed and placed into applications.
Typical goals for a control such as the demo PhotoStrip control could be to:
• Respond to touch messages
• Add constrained manipulations for panning
• Add support for dragging in / out of the control
• Add an AIP surface
• Flesh out the overlay and gallery
The following image shows the PhotoStrip control in the demo application.
There are PhotoStrip controls on the top and the bottom of the screen. Sliding along the control will move the control. Dragging images from the bottom of the control up will cause them to appear in the Gallery.
Demo Overview
The PhotoStrip control is encapsulated into its own HWND. This is a problem if you want to have many controls working well together because Windows Touch messages are only sent to a single window, the first window that receives the touch down message. To work around this, an overlay window takes all the input and maps it to the correct control as appropriate based on hit detection. The following diagram outlines this hierarchy of Windows.
Given that window hierarchy with the overlay, the following shows how data would flow through the control.
In the message flow, a WM_TOUCH message is generated by the input hardware. This message will be received by the overlay HWND. The overlay HWND then performs hit detection and sends the message to the correct child HWND (the PhotoStrip window, for example). If the user were dragging an image to the gallery, the PhotoStrip would then send a message to the application which tells it that the user dragged a photo to the gallery. The application then tells the gallery to display the photo in the gallery.
The following image illustrates the controls of the Demo application in a screenshot of the demo running.
There are PhotoStrip controls on the top and bottom of the application and the Gallery control is in the center of the application layout.
Programming Details
Handling messaging to the Overlay control
First off, the application creates the overlay window and then creates the controls for the gallery and the PhotoStrips. These controls are then registered with the overlay.
The following code shows how this is done in code.
HRESULT InitWindow( HINSTANCE hInstance )
{
(…)
if( SUCCEEDED( hr ) )
{
// Create window
hWnd = CreateWindowEx(0, WINDOW_CLASS_NAME, L"CollageSample",
WS_OVERLAPPEDWINDOW, CW_USEDEFAULT, CW_USEDEFAULT,
CW_USEDEFAULT, CW_USEDEFAULT, NULL, NULL, hInstance, 0);
if (!(hWnd))
{
hr = E_FAIL;
}
}
if( SUCCEEDED( hr ) )
{
ShowWindow( hWnd, SW_SHOWMAXIMIZED );
DisableTabFeedback(hWnd);
}
if (SUCCEEDED( hr ))
{
GetClientRect((hWnd), &rClient);
iWidth = rClient.right - rClient.left;
iHeight = rClient.bottom - rClient.top;
iTopHeight = (INT)(iHeight * TOP_HEIGHT_PERCENTAGE);
iBottomHeight = (INT)(iHeight * BOTTOM_HEIGHT_PERCENTAGE);
hr = Overlay::CreateOverlay(hInstance, hWnd, 0, 0,
iWidth, iHeight, &g_pOverlay);
}
if (SUCCEEDED( hr ))
{
hr = Photostrip::CreatePhotostrip(hInstance, (hWnd),
0, 0,
iWidth, iTopHeight,
TOP_PHOTOSTRIP_DIRECTORY,
Photostrip::Top,
&g_pStripTop);
}
if (SUCCEEDED( hr ))
{
hr = Photostrip::CreatePhotostrip(hInstance, (hWnd),
0, rClient.bottom - iBottomHeight,
iWidth, iBottomHeight,
BOTTOM_PHOTOSTRIP_DIRECTORY,
Photostrip::Bottom,
&g_pStripBottom);
}
if (SUCCEEDED( hr ))
{
hr = Gallery::CreateGallery(hInstance, hWnd, 0, iTopHeight,
iWidth, (iHeight - iTopHeight - iBottomHeight), &g_pGallery);
}
if (SUCCEEDED( hr ))
{
g_pOverlay->RegisterTarget((ITouchTarget*)g_pGallery);
g_pOverlay->RegisterTarget((ITouchTarget*)g_pStripBottom);
g_pOverlay->RegisterTarget((ITouchTarget*)g_pStripTop);
}
When WM_TOUCH messages are generated, the overlay window will receive them in its WndProc method. The overlay converts the message coordinates from screen coordinates to client coordinates and keeps track of the messages. The overlay keeps track of particular inputs and then gives capture to a particular control.
The following code shows how the overlay maps inputs to the appropriate child control.
LRESULT CALLBACK Overlay::S_OverlayWndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
{
//reroute WM_TOUCH and mouse messages
//if it's a down event, rehit test and set capture,
//else, route the event to the target currently capturing the cursor
LRESULT result = 0;
INT iNumContacts;
PTOUCHINPUT pInputs;
HTOUCHINPUT hInput;
POINT ptInputs;
ITouchTarget *pTarget = NULL;
Overlay* pOverlay;
pOverlay = (Overlay*)GetWindowLongPtr(hWnd, 0);
if (pOverlay != NULL)
{
switch(msg)
{
case WM_TOUCH:
iNumContacts = LOWORD(wParam);
hInput = (HTOUCHINPUT)lParam;
pInputs = new (std::nothrow) TOUCHINPUT[iNumContacts];
if(pInputs != NULL)
{
if(GetTouchInputInfo(hInput, iNumContacts, pInputs, sizeof(TOUCHINPUT)))
{
for(int i = 0; i < iNumContacts; i++)
{
//track the number of contacts
if (pInputs[i].dwFlags & TOUCHEVENTF_DOWN)
{
pOverlay->m_ucContacts++;
}
else if (pInputs[i].dwFlags & TOUCHEVENTF_UP)
{
pOverlay->m_ucContacts--;
}
// Bring touch input info into client coordinates
ptInputs.x = pInputs[i].x/100;
ptInputs.y = pInputs[i].y/100;
ScreenToClient(hWnd, &ptInputs);
pInputs[i].x = (LONG)(ptInputs.x);
pInputs[i].y = (LONG)(ptInputs.y);
//hit test and set capture on down
if (pInputs[i].dwFlags & TOUCHEVENTF_DOWN)
{
pOverlay-> m_pDispatch->CaptureCursor(pInputs[i].dwID,
(FLOAT)pInputs[i].x, (FLOAT)pInputs[i].y, &pTarget);
}
//if anything has capture set for this cursor, route the event to it
if (pOverlay->m_pDispatch->GetCapturingTarget(pInputs[i].dwID, &pTarget))
{
pTarget->ProcessTouchInput(&pInputs[i]);
}
}
}
delete [] pInputs;
}
CloseTouchInputHandle(hInput);
break;
case WM_LBUTTONUP:
ReleaseCapture();
pOverlay->m_fIsMouseDown = FALSE;
pOverlay->ProcessMouse(msg, wParam, lParam);
break;
case WM_MOUSEMOVE:
pOverlay->ProcessMouse(msg, wParam, lParam);
break;
case WM_LBUTTONDOWN:
SetCapture(hWnd);
pOverlay->m_fIsMouseDown = TRUE;
pOverlay->ProcessMouse(msg, wParam, lParam);
break;
case WM_DESTROY:
SetWindowLongPtr(hWnd, 0, 0);
PostQuitMessage(0);
delete pOverlay;
break;
default:
result = DefWindowProc(hWnd, msg, wParam, lParam);
}
}
else
{
result = DefWindowProc(hWnd, msg, wParam, lParam);
}
return result;
}
Each of the controls handles the touch messages in a similar manner in the ProcessTouchInput method which will cause _ManipulationEvents to be raised on the control.
Handling Touch input for the PhotoStrip
The PhotoStrip itself implements the InertiaObj utility interface to enable inertia features. The following diagram illustrates the WM_TOUCH input mapping from the overlay window to the PhotoStrip control and its photos.
When capture goes to the control, the WM_TOUCH messages are sent to a ManipulationEvents interface which will raise manipulation events on that interface. Horizontal movement within the control will manipulate all of the photos simultaneously. Hit testing within the PhotoStrip control is used to send input to the appropriate photo because the photos themselves can be manipulated vertically within the control. Once the PhotoStrip control has finished interpreting input, that control releases capture for the control.
The following code shows how input is mapped to manipulations on the ManipulationDelta handler.
HRESULT STDMETHODCALLTYPE Photostrip::ManipulationDelta(
FLOAT /*x*/,
FLOAT /*y*/,
FLOAT translationDeltaX,
FLOAT /*translationDeltaY*/,
FLOAT /*scaleDelta*/,
FLOAT /*expansionDelta*/,
FLOAT /*rotationDelta*/,
FLOAT /*cumulativeTranslationX*/,
FLOAT /*cumulativeTranslationY*/,
FLOAT /*cumulativeScale*/,
FLOAT /*cumulativeExpansion*/,
FLOAT /*cumulativeRotation*/)
{
HRESULT hr = S_OK;
//bound the move to the area around our photos
FLOAT fpTotalWidth = -INTERNAL_MARGIN, fpNewXOffset;
std::list<Photo*>::iterator it;
for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++)
{
fpTotalWidth += (*it)->GetWidth() + INTERNAL_MARGIN;
}
fpTotalWidth = max(0, fpTotalWidth); //handle no photos
fpNewXOffset = m_fpXOffset + translationDeltaX;
FLOAT fpXLowerBound = -fpTotalWidth + m_nWidth * (1-SIDE_MARGIN_PERCENTAGE);
FLOAT fpXUpperBound = m_nWidth * SIDE_MARGIN_PERCENTAGE;
fpNewXOffset = min(fpXUpperBound, fpNewXOffset);
fpNewXOffset = max(fpXLowerBound, fpNewXOffset);
translationDeltaX = fpNewXOffset - m_fpXOffset;
//move the photos
m_fpXOffset += translationDeltaX;
for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++)
{
(*it)->Translate(translationDeltaX, 0);
}
return hr;
}
The following code shows how the PhotoStrip uses manipulations to move photos horizontally along the control in the ManipulationDelta event handler.
//move the photos
m_fpXOffset += translationDeltaX;
for (it = m_lPhotos.begin(); it != m_lPhotos.end(); it++)
{
(*it)->Translate(translationDeltaX, 0);
}
The following code shows how photos in the PhotoStrip control handle manipulations and are constrained to vertical manipulations.
HRESULT STDMETHODCALLTYPE ConstrainedPhoto::ManipulationDelta(
FLOAT /*x*/,
FLOAT /*y*/,
FLOAT /*translationDeltaX*/,
FLOAT translationDeltaY,
FLOAT /*scaleDelta*/,
FLOAT /*expansionDelta*/,
FLOAT /*rotationDelta*/,
FLOAT /*cumulativeTranslationX*/,
FLOAT /*cumulativeTranslationY*/,
FLOAT /*cumulativeScale*/,
FLOAT /*cumulativeExpansion*/,
FLOAT /*cumulativeRotation*/)
{
HRESULT hr = S_OK;
Translate(0.0f, translationDeltaY);
return hr;
}
Note how the boundary checks from the photo object will notify the application that the photo has reached the boundary. A fake touch up message is sent up and capture from the PhotoStrip is released.
The following code shows how photos release capture.
case PS_PHOTO_BOUNDARY:
if (pStrip != NULL)
{
Photo *pPhoto = (Photo*)lParam;
TOUCHINPUT tUp = pPhoto->GetLastTouchInput();
pStrip->m_pDispatch->ReleaseCapture(tUp.dwID);
//since we are removing the photo from the event stream by releasing
//capture, it will never see an up for this cursor
//fake an up to the photo so it's view of the event stream stays consistent
tUp.dwFlags &= ~TOUCHEVENTF_DOWN;
tUp.dwFlags &= ~TOUCHEVENTF_MOVE;
tUp.dwFlags |= TOUCHEVENTF_UP;
pPhoto->ProcessTouchInput(&tUp);
//let out parent know that the user dragged a photo off
SendMessage(GetParent(hWnd), PS_PHOTOSTRIP_BOUNDARY, (WPARAM)pStrip, lParam);
}
break;
The message that is generated to the parent window contains a pointer to the photo object which can then be used rendered the photo to the gallery control.
The Collage Control
When a photo is moved to the collage window, the collage uses the message to create an image that it displays. The following code shows how this is done.
case PS_PHOTOSTRIP_BOUNDARY:
//if a photo was dragged off a photostrip, we want to -
// -remove the photostrip's capture of the cursor that dragged the photo off
// -set the gallery to capture that cursor, so the photo can be dragged around
// immedietely
// -load the photo that was dragged off the stip into the gallery
pPhoto = (Photo*)lParam;
pStrip = (Photostrip*)wParam;
tInput = pPhoto->GetLastTouchInput();
g_pOverlay->ReleaseCapturedCursor(tInput.dwID);
//since we released capture, the strip won't ever get an up for this event stream
//fake an up to the target so it stays consistent
tUp = tInput;
tUp.dwFlags &= ~TOUCHEVENTF_DOWN;
tUp.dwFlags &= ~TOUCHEVENTF_MOVE;
tUp.dwFlags |= TOUCHEVENTF_UP;
pStrip->ProcessTouchInput(&tUp);
//translate into the parent window coordinates
GetWindowRect(pPhoto->GetHWnd(), &stripWnd);
GetWindowRect(g_pGallery->GetHWnd(), &galleryWnd);
tInput.x += stripWnd.left;
tInput.y += stripWnd.top;
//calculate the photo size and position
fpPhotoHeight = (FLOAT)(stripWnd.bottom - stripWnd.top)
* (1 - INTERNAL_Y_MARGIN_PERCENTAGE)
* PHOTO_SIZE_MULTIPLIER;
fpPhotoYPos = (FLOAT)(tInput.y - galleryWnd.top);
fpPhotoYPos -= ((stripWnd.bottom - stripWnd.top) * INTERNAL_Y_MARGIN_PERCENTAGE);
//load the photo into the gallery
g_pGallery->LoadPhoto(pPhoto->GetPhotoURI(),
(FLOAT)tInput.x, fpPhotoYPos, fpPhotoHeight);
//direct the input to the new photo
g_pOverlay->SetCursorCapture(g_pGallery, tInput);
break;
When a photo reaches a boundary in the collage window, the collage will then remove the photo from the list of objects that it is tracking and will release capture of touch input. The following code shows how this is done in the case of photos hitting a boundary.
case PS_PHOTO_BOUNDARY_INERTIA:
case PS_PHOTO_BOUNDARY:
if (pCollage != NULL)
{
//mark the photo for deletion, it will be removed the next time the timer fires
pPhoto = (Photo*)lParam;
if (pPhoto != NULL)
{
pCollage->m_lPhotosToDelete.remove(pPhoto); //make sure we don't add it twice
pCollage->m_lPhotosToDelete.push_front(pPhoto);
}
}
break;
The following code shows how this is done in the case of a boundary being passed during inertia.
case WM_TIMER:
if (pCollage != NULL)
{
//process inertia
for (it = pCollage->m_lPhotos.begin(); it != pCollage->m_lPhotos.end(); it++)
{
(*it)->ProcessTimer();
}
//processing the timers might have caused a boundry event, in
//which case we would need to clean up photos that passed the boundry
for (it = pCollage->m_lPhotosToDelete.begin();
it != pCollage->m_lPhotosToDelete.end(); it++)
{
//only remove the photo if it's still past the bounds. as we have capturing,
//the user could have dragged the photo off the window then back on
if ((*it)->IsPastBounds())
{
//release any current mapping
DWORD dwID = 0;
while(pCollage->m_pDispatch->GetCursor(*it, &dwID))
{
pCollage->m_pDispatch->ReleaseCapture(dwID);
}
pCollage->m_lPhotos.remove((Photo*)(*it));
pCollage->m_pDispatch->UnregisterTouchTarget((*it));
(*it)->CleanUp();
}
}
pCollage->m_lPhotosToDelete.clear();
pCollage->Render();
}
break;
Conclusions
Creating a control framework for complex Windows Touch applications is the best way to deliver an experience with reusable components. A newer version of this document will be released online at https://code.msdn.microsoft.com/WinTouchPhotostrip later this month (February) so keep an eye on the @WinDevs twitter account if you are looking forward to the update. Apologies for no colorization in the C++ code, I’m working on potential solutions that will work with this blog platform.