Gaze Interaction Library

Microsoft announced native support for eye tracking in Windows in the Windows 10 Fall Creators Update. In the Windows 10 April 2018 update, Microsoft added developer support by releasing Windows Gaze Input APIs to build UWP applications that can interact with gaze input and eye trackers.

This Gaze Interaction Library is built on top of the Windows Gaze Input APIs and provides a set of developer helper classes to more easily enable UWP applications to respond to where the user is looking on the screen. This library is intended to abstract away some of the complexities of dealing with the raw stream of gaze input coming from the eye tracking device exposed through the Windows APIs.

Prerequisites

In order to use the Windows 10 Gaze Input APIs or this Gaze Interaction Library, you need to have the following:

• A device with the Windows 10 April 2018 Update
• A supported eye tracker, like the Tobii EyeX 4C

To use the gaze input APIs, you must register the gazeInput capability in the app manifest. Add the following DeviceCapability to the Capabilities node in the Package.appxmanifest file of your project (you can do this by checking Gaze Input in the Capabilities tab in Visual Studio when you have Package.appxmanifest open in the App Manifest Designer):

<Capabilities>
   <DeviceCapability Name="gazeInput" />
</Capabilities>

Supported Features

The Gaze Interaction Library currently supports the following features:

• Dwell based activation of XAML button and toggle button controls
• Enabling gaze interactions for the entire XAML page or for a portion of it (like a single control)
• Customizing the dwell times associated with specific controls
• Controlling repetition of the dwell invocation of controls

Gaze Concepts

Below are a few eye gaze related concepts that are helpful in order to better understand the rest of the document:

• Saccade. A saccade is a rapid movement of the eyes from one fixation point to another. Our eyes alternate between fixations and saccades.
• Fixation. A fixation is the collection of gaze points on a single location of the screen for a relatively short amount of time (roughly around 200ms). This happens after a saccade when the eye comes to rest upon an object and it comes into sharp focus.
• Dwell. A dwell is the conscious fixation by the user for a duration greater than the fixation time. This mechanism is typically used to identify user intent when the user is using only their eyes as an input method. The dwell time duration can be application and control dependent.
• Enter/Exit. These are states and properties specific to this API to help manage gaze related interactions and refer to the time elapsed since the first recorded gaze sample and the last recorded gaze sample on a particular control (Button, ToggleButton etc.)

The Gaze Interaction Library enables dwell based gaze interactions on the page by reading the data from the eye tracker over the page invoking specific controls when the user's gaze dwells on a control for a specific time. The application can configure this time based on its usage scenario.

PointerState Properties

The Windows Gaze Input APIs delivers a stream of timestamped gaze point coordinates of the user's gaze location on the screen. This library aggregates these samples over each control and converts the stream into gaze events. Corresponding to these events, are the following states:

GazeInput Properties

If the XAML page is enabled for gaze based interactions, the visibility and size of the gaze cursor, and the timings associated with the states above can be configured using the properties below:

GazeElement Events

Add description here.

StateChangedEventArgs Properties

This parameter is passed to the StateChanged event which is fired when the PointerState changes.

DwellProgressEventArgs Properties

This parameter is passed to the event handling the DwellProgressFeedback event.

DwellInvokedRoutedEventArgs Properties

This parameter is passed to the GazeElement::Invoked event and allows the application to prevent default invocation when the user dwells on a control.

Examples

To enable gaze interactions on the entire XAML page

Add the following lines to your Page element to enable gaze interaction on the entire page.

    xmlns:gaze="using:Microsoft.Toolkit.Uwp.Input.GazeInteraction"
    gaze:GazeInput.Interaction="Enabled"

For e.g.

    <Page
    x:Class="UwpApp.MainPage"
    xmlns="https://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="https://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:d="https://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="https://schemas.openxmlformats.org/markup-compatibility/2006"
    xmlns:gaze="using:Microsoft.Toolkit.Uwp.Input.GazeInteraction"
    gaze:GazeInput.Interaction="Enabled"
    mc:Ignorable="d">

To enable gaze interactions on a portion of the page

Gaze interaction can also be enabled only on a portion of the page by adding the same attributes to any XAML element on the page.

To enable only a subset of the page, e.g. just one Grid on the page:

    <Grid gaze:GazeInput.Interaction="Enabled">
        <Button Content="Click Me" />
    <Grid />

In the above example, the button will be clicked when the user looks at the button in the grid for a period equal to the default dwell time.

To change the dwell time for a control

The code below sets the Dwell time duration for the button to be 500ms. This means the button will be clicked 500ms after the control enters the Fixation state. (See PointerState for details)

    <Button Content="Click Me" gaze:GazeInput.DwellDuration="00:00:00.500">

Animations

This library provides a default animation of a shriniking rectangle over the control to indicate progress towards a dwell event. The style of animation can be customized in two ways:

• If you wish to retain the animation style, but change the colors used:
  • Set the GazeInput.DwellFeedbackEnterBrush property to add a rectangle over controls immediately gaze is detected on them
  • Change the GazeInput.DwellFeedbackProgressBrush property to change the color of the progress rectangle
  • Change the GazeInput.DwellFeedbackCompletedBrush property to change the color of the completed brush
• If you wish to completely disable or override the default animation, add an event handler for DwellProgressFeedback on GazeElement and set Handled to True as follows

<Button Content="Click Me">
    <gaze:GazeInput.GazeElement>
        <gaze:GazeElement DwellProgressFeedback="OnInvokeProgress" />
    </gaze:GazeInput.GazeElement>
</Button>

private void OnInvokeProgress(object sender, DwellProgressEventArgs e)
{
    e.Handled = true;
}

Sample Project

GazeInteractionPage. You can see this in action in the Windows Community Toolkit Sample App.

Requirements

API

• Gaze Interaction Library source code

Related Topics

• Windows 10 Gaze Input APIs