Pen interactions and Windows Ink in Windows apps

Surface Pen (available for purchase at the Microsoft Store).

Overview

Optimize your Windows app for pen input to provide both standard pointer device functionality and the best Windows Ink experience for your users.

The Windows Ink platform, together with a pen device, provides a natural way to create digital handwritten notes, drawings, and annotations. The platform supports capturing digitizer input as ink data, generating ink data, managing ink data, rendering ink data as ink strokes on the output device, and converting ink to text through handwriting recognition.

In addition to capturing the basic position and movement of the pen as the user writes or draws, your app can also track and collect the varying amounts of pressure used throughout a stroke. This information, along with settings for pen tip shape, size, and rotation, ink color, and purpose (plain ink, erasing, highlighting, and selecting), enables you to provide user experiences that closely resemble writing or drawing on paper with a pen, pencil, or brush.

The ink platform is very flexible. It is designed to support various levels of functionality, depending on your requirements.

For Windows Ink UX guidelines, see Inking controls.

Components of the Windows Ink platform

Basic inking with InkCanvas

To add basic inking functionality, just place an InkCanvas UWP platform control on the appropriate page in your app.

By default, the InkCanvas supports ink input only from a pen. The input is either rendered as an ink stroke using default settings for color and thickness (a black ballpoint pen with a thickness of 2 pixels), or treated as a stroke eraser (when input is from an eraser tip or the pen tip modified with an erase button).

In this example, an InkCanvas overlays a background image.

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.RowDefinitions>
        <RowDefinition Height="Auto"/>
        <RowDefinition Height="*"/>
    </Grid.RowDefinitions>
    <StackPanel x:Name="HeaderPanel" Orientation="Horizontal" Grid.Row="0">
        <TextBlock x:Name="Header"
                   Text="Basic ink sample"
                   Style="{ThemeResource HeaderTextBlockStyle}"
                   Margin="10,0,0,0" />            
    </StackPanel>
    <Grid Grid.Row="1">
        <Image Source="Assets\StoreLogo.png" />
        <InkCanvas x:Name="inkCanvas" />
    </Grid>
</Grid>

This series of images shows how pen input is rendered by this InkCanvas control.

The blank InkCanvas with a background image.	The InkCanvas with ink strokes.	The InkCanvas with one stroke erased (note how erase operates on an entire stroke, not a portion).

The inking functionality supported by the InkCanvas control is provided by a code-behind object called the InkPresenter.

For basic inking, you don't have to concern yourself with the InkPresenter. However, to customize and configure inking behavior on the InkCanvas, you must access its corresponding InkPresenter object.

Basic customization with InkPresenter

An InkPresenter object is instantiated with each InkCanvas control.

Along with providing all default inking behaviors of its corresponding InkCanvas control, the InkPresenter provides a comprehensive set of APIs for additional stroke customization and finer-grained management of the pen input (standard and modified). This includes stroke properties, supported input device types, and whether input is processed by the object or passed to the app for processing.

By default, ink is supported for pen input only. Here, we configure the InkPresenter to interpret input data from both pen and mouse as ink strokes. We also set some initial ink stroke attributes used for rendering strokes to the InkCanvas.

To enable mouse and touch inking, set the InputDeviceTypes property of the InkPresenter to the combination of CoreInputDeviceTypes values that you want.

public MainPage()
{
    this.InitializeComponent();

    // Set supported inking device types.
    inkCanvas.InkPresenter.InputDeviceTypes =
        Windows.UI.Core.CoreInputDeviceTypes.Mouse |
        Windows.UI.Core.CoreInputDeviceTypes.Pen;

    // Set initial ink stroke attributes.
    InkDrawingAttributes drawingAttributes = new InkDrawingAttributes();
    drawingAttributes.Color = Windows.UI.Colors.Black;
    drawingAttributes.IgnorePressure = false;
    drawingAttributes.FitToCurve = true;
    inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttributes);
}

Ink stroke attributes can be set dynamically to accommodate user preferences or app requirements.

Here, we let a user choose from a list of ink colors.

<Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.RowDefinitions>
        <RowDefinition Height="Auto"/>
        <RowDefinition Height="*"/>
    </Grid.RowDefinitions>
    <StackPanel x:Name="HeaderPanel" Orientation="Horizontal" Grid.Row="0">
        <TextBlock x:Name="Header"
                   Text="Basic ink customization sample"
                   VerticalAlignment="Center"
                   Style="{ThemeResource HeaderTextBlockStyle}"
                   Margin="10,0,0,0" />
        <TextBlock Text="Color:"
                   Style="{StaticResource SubheaderTextBlockStyle}"
                   VerticalAlignment="Center"
                   Margin="50,0,10,0"/>
        <ComboBox x:Name="PenColor"
                  VerticalAlignment="Center"
                  SelectedIndex="0"
                  SelectionChanged="OnPenColorChanged">
            <ComboBoxItem Content="Black"/>
            <ComboBoxItem Content="Red"/>
        </ComboBox>
    </StackPanel>
    <Grid Grid.Row="1">
        <Image Source="Assets\StoreLogo.png" />
        <InkCanvas x:Name="inkCanvas" />
    </Grid>
</Grid>

We then handle changes to the selected color and update the ink stroke attributes accordingly.

// Update ink stroke color for new strokes.
private void OnPenColorChanged(object sender, SelectionChangedEventArgs e)
{
    if (inkCanvas != null)
    {
        InkDrawingAttributes drawingAttributes =
            inkCanvas.InkPresenter.CopyDefaultDrawingAttributes();

        string value = ((ComboBoxItem)PenColor.SelectedItem).Content.ToString();

        switch (value)
        {
            case "Black":
                drawingAttributes.Color = Windows.UI.Colors.Black;
                break;
            case "Red":
                drawingAttributes.Color = Windows.UI.Colors.Red;
                break;
            default:
                drawingAttributes.Color = Windows.UI.Colors.Black;
                break;
        };

        inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttributes);
    }
}

These images shows how pen input is processed and customized by the InkPresenter.

To provide functionality beyond inking and erasing, such as stroke selection, your app must identify specific input for the InkPresenter to pass through unprocessed for handling by your app.

Pass-through input for advanced processing

By default, InkPresenter processes all input as either an ink stroke or an erase stroke, including input modified by a secondary hardware affordance such as a pen barrel button, a right mouse button, or similar. However, users typically expect some additional functionality or modified behavior with these secondary affordances.

In some cases, you might also need to expose additional functionality for pens without secondary affordances (functionality not usually associated with the pen tip), other input device types, or some type of modified behavior based on a user selection in your app's UI.

To support this, InkPresenter can be configured to leave specific input unprocessed. This unprocessed input is then passed through to your app for processing.

Example - Use unprocessed input to implement stroke selection

The Windows Ink platform does not provide built-in support for actions that require modified input, such as stroke selection. To support features like this, you must provide a custom solution in your apps.

The following code example (all code is in the MainPage.xaml and MainPage.xaml.cs files) steps through how to enable stroke selection when input is modified with a pen barrel button (or right mouse button).

1. First, we set up the UI in MainPage.xaml.

   Here, we add a canvas (below the InkCanvas) to draw the selection stroke. Using a separate layer to draw the selection stroke leaves the InkCanvas and its content untouched.

   

     <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
       <Grid.RowDefinitions>
         <RowDefinition Height="Auto"/>
         <RowDefinition Height="*"/>
       </Grid.RowDefinitions>
       <StackPanel x:Name="HeaderPanel" Orientation="Horizontal" Grid.Row="0">
         <TextBlock x:Name="Header"
           Text="Advanced ink customization sample"
           VerticalAlignment="Center"
           Style="{ThemeResource HeaderTextBlockStyle}"
           Margin="10,0,0,0" />
       </StackPanel>
       <Grid Grid.Row="1">
         <!-- Canvas for displaying selection UI. -->
         <Canvas x:Name="selectionCanvas"/>
         <!-- Inking area -->
         <InkCanvas x:Name="inkCanvas"/>
       </Grid>
     </Grid>
   

2. In MainPage.xaml.cs, we declare a couple of global variables for keeping references to aspects of the selection UI. Specifically, the selection lasso stroke and the bounding rectangle that highlights the selected strokes.

     // Stroke selection tool.
     private Polyline lasso;
     // Stroke selection area.
     private Rect boundingRect;
   

3. Next, we configure the InkPresenter to interpret input data from both pen and mouse as ink strokes, and set some initial ink stroke attributes used for rendering strokes to the InkCanvas.

   Most importantly, we use the InputProcessingConfiguration property of the InkPresenter to indicate that any modified input should be processed by the app. Modified input is specified by assigning InputProcessingConfiguration.RightDragAction a value of InkInputRightDragAction.LeaveUnprocessed. When this value is set, the InkPresenter passes through to the InkUnprocessedInput class, a set of pointer events for you to handle.

   We assign listeners for the unprocessed PointerPressed, PointerMoved, and PointerReleased events passed through by the InkPresenter. All selection functionality is implemented in the handlers for these events.

   Finally, we assign listeners for the StrokeStarted and StrokesErased events of the InkPresenter. We use the handlers for these events to clean up the selection UI if a new stroke is started or an existing stroke is erased.

   

     public MainPage()
     {
       this.InitializeComponent();
   
       // Set supported inking device types.
       inkCanvas.InkPresenter.InputDeviceTypes =
         Windows.UI.Core.CoreInputDeviceTypes.Mouse |
         Windows.UI.Core.CoreInputDeviceTypes.Pen;
   
       // Set initial ink stroke attributes.
       InkDrawingAttributes drawingAttributes = new InkDrawingAttributes();
       drawingAttributes.Color = Windows.UI.Colors.Black;
       drawingAttributes.IgnorePressure = false;
       drawingAttributes.FitToCurve = true;
       inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttributes);
   
       // By default, the InkPresenter processes input modified by
       // a secondary affordance (pen barrel button, right mouse
       // button, or similar) as ink.
       // To pass through modified input to the app for custom processing
       // on the app UI thread instead of the background ink thread, set
       // InputProcessingConfiguration.RightDragAction to LeaveUnprocessed.
       inkCanvas.InkPresenter.InputProcessingConfiguration.RightDragAction =
           InkInputRightDragAction.LeaveUnprocessed;
   
       // Listen for unprocessed pointer events from modified input.
       // The input is used to provide selection functionality.
       inkCanvas.InkPresenter.UnprocessedInput.PointerPressed +=
           UnprocessedInput_PointerPressed;
       inkCanvas.InkPresenter.UnprocessedInput.PointerMoved +=
           UnprocessedInput_PointerMoved;
       inkCanvas.InkPresenter.UnprocessedInput.PointerReleased +=
           UnprocessedInput_PointerReleased;
   
       // Listen for new ink or erase strokes to clean up selection UI.
       inkCanvas.InkPresenter.StrokeInput.StrokeStarted +=
           StrokeInput_StrokeStarted;
       inkCanvas.InkPresenter.StrokesErased +=
           InkPresenter_StrokesErased;
     }
   

4. We then define handlers for the unprocessed PointerPressed, PointerMoved, and PointerReleased events passed through by the InkPresenter.

   All selection functionality is implemented in these handlers, including the lasso stroke and the bounding rectangle.

   

     // Handle unprocessed pointer events from modified input.
     // The input is used to provide selection functionality.
     // Selection UI is drawn on a canvas under the InkCanvas.
     private void UnprocessedInput_PointerPressed(
       InkUnprocessedInput sender, PointerEventArgs args)
     {
       // Initialize a selection lasso.
       lasso = new Polyline()
       {
           Stroke = new SolidColorBrush(Windows.UI.Colors.Blue),
           StrokeThickness = 1,
           StrokeDashArray = new DoubleCollection() { 5, 2 },
           };
   
           lasso.Points.Add(args.CurrentPoint.RawPosition);
   
           selectionCanvas.Children.Add(lasso);
       }
   
       private void UnprocessedInput_PointerMoved(
         InkUnprocessedInput sender, PointerEventArgs args)
       {
         // Add a point to the lasso Polyline object.
         lasso.Points.Add(args.CurrentPoint.RawPosition);
       }
   
       private void UnprocessedInput_PointerReleased(
         InkUnprocessedInput sender, PointerEventArgs args)
       {
         // Add the final point to the Polyline object and
         // select strokes within the lasso area.
         // Draw a bounding box on the selection canvas
         // around the selected ink strokes.
         lasso.Points.Add(args.CurrentPoint.RawPosition);
   
         boundingRect =
           inkCanvas.InkPresenter.StrokeContainer.SelectWithPolyLine(
             lasso.Points);
   
         DrawBoundingRect();
       }
   

5. To conclude the PointerReleased event handler, we clear the selection layer of all content (the lasso stroke) and then draw a single bounding rectangle around the ink strokes encompassed by the lasso area.

   

     // Draw a bounding rectangle, on the selection canvas, encompassing
     // all ink strokes within the lasso area.
     private void DrawBoundingRect()
     {
       // Clear all existing content from the selection canvas.
       selectionCanvas.Children.Clear();
   
       // Draw a bounding rectangle only if there are ink strokes
       // within the lasso area.
       if (!((boundingRect.Width == 0) ||
         (boundingRect.Height == 0) ||
         boundingRect.IsEmpty))
         {
           var rectangle = new Rectangle()
           {
             Stroke = new SolidColorBrush(Windows.UI.Colors.Blue),
               StrokeThickness = 1,
               StrokeDashArray = new DoubleCollection() { 5, 2 },
               Width = boundingRect.Width,
               Height = boundingRect.Height
           };
   
           Canvas.SetLeft(rectangle, boundingRect.X);
           Canvas.SetTop(rectangle, boundingRect.Y);
   
           selectionCanvas.Children.Add(rectangle);
         }
       }
   

6. Finally, we define handlers for the StrokeStarted and StrokesErased InkPresenter events.

   These both just call the same cleanup function to clear the current selection whenever a new stroke is detected.

     // Handle new ink or erase strokes to clean up selection UI.
     private void StrokeInput_StrokeStarted(
       InkStrokeInput sender, Windows.UI.Core.PointerEventArgs args)
     {
       ClearSelection();
     }
   
     private void InkPresenter_StrokesErased(
       InkPresenter sender, InkStrokesErasedEventArgs args)
     {
       ClearSelection();
     }
   

7. Here's the function to remove all selection UI from the selection canvas when a new stroke is started or an existing stroke is erased.

     // Clean up selection UI.
     private void ClearSelection()
     {
       var strokes = inkCanvas.InkPresenter.StrokeContainer.GetStrokes();
       foreach (var stroke in strokes)
       {
         stroke.Selected = false;
       }
       ClearDrawnBoundingRect();
      }
   
     private void ClearDrawnBoundingRect()
     {
       if (selectionCanvas.Children.Any())
       {
         selectionCanvas.Children.Clear();
         boundingRect = Rect.Empty;
       }
     }
   

Custom ink rendering

By default, ink input is processed on a low-latency background thread and rendered in-progress, or "wet", as it is drawn. When the stroke is completed (pen or finger lifted, or mouse button released), the stroke is processed on the UI thread and rendered "dry" to the InkCanvas layer (above the application content and replacing the wet ink).

You can override this default behavior and completely control the inking experience by "custom drying" the wet ink strokes. While the default behavior is typically sufficient for most applications, there are a few cases where custom drying might be required, these include:

• More efficient management of large, or complex, collections of ink strokes
• More efficient panning and zooming support on large ink canvases
• Interleaving ink and other objects, such as shapes or text, while maintaining z-order
• Drying and converting ink synchronously into a DirectX shape (for example, a straight line or shape rasterized and integrated into application content instead of as a separate InkCanvas layer).

Custom drying requires an IInkD2DRenderer object to manage the ink input and render it to the Direct2D device context of your Universal Windows app, instead of the default InkCanvas control.

By calling ActivateCustomDrying (before the InkCanvas is loaded), an app creates an InkSynchronizer object to customize how an ink stroke is rendered dry to a SurfaceImageSource or VirtualSurfaceImageSource.

Both SurfaceImageSource and VirtualSurfaceImageSource provide a DirectX shared surface for your app to draw into and compose into your application's content, although VSIS provides a virtual surface that’s larger than the screen for performant panning and zooming. Because visual updates to these surfaces are synchronized with the XAML UI thread, when ink is rendered to either, the wet ink can be removed from the InkCanvas simultaneously.

You can also custom dry ink to a SwapChainPanel, but synchronization with the UI thread is not guaranteed and there might be a delay between when the ink is rendered to your SwapChainPanel and when ink is removed from the InkCanvas.

For a full example of this functionality, see the Complex ink sample.

Other articles in this section

Related articles

• Get started: Support ink in your Windows app
• Handle pointer input
• Identify input devices
• Ink Serialized Format (ISF) Specification.

APIs

• Windows.Devices.Input
• Windows.UI.Input.Inking
• Windows.UI.Input.Inking.Core

Samples

• Get Started Tutorial: Support ink in your Windows app
• Simple ink sample (C#/C++)
• Complex ink sample (C++)
• Ink sample (JavaScript)
• Coloring book sample
• Family notes sample
• Basic input sample
• Low latency input sample
• User interaction mode sample
• Focus visuals sample

Archive Samples

• Input: Device capabilities sample
• Input: XAML user input events sample
• XAML scrolling, panning, and zooming sample
• Input: Gestures and manipulations with GestureRecognizer