Kinect and UWP: new options in the Windows 10 Anniversary Update

  • Article

UWP apps today cannot access most of the data from a Kinect sensor. This post provides an overview of what we plan to deliver in the near future to begin bringing Kinect for Windows into the Universal Windows Platform (UWP) world.

With the Windows 10 Anniversary Update and a Kinect driver update, Kinect developers will be able to read RGB, infrared (IR), and depth data from the Kinect sensor. The Kinect sensor’s microphone array is already visible to the UWP audio and speech recognition APIs today (although not all features from the Kinect for Windows SDK are accessible to UWP APIs)—this remains unchanged.

Skeleton data is important to many Kinect apps. Through a supplemental SDK that will be available later this year, UWP developers will have access to skeletal poses. With the new capabilities in the Windows 10 Anniversary Update, developers can also use the existing Windows.Media.FaceAnalysis APIs with Kinect to detect and track faces.

Existing Kinect Win32 apps and SDK capabilities will not be affected by this new UWP support.

Details: how it works

Today, there are several sensors that can provide rich correlated data, such as RGB, IR, or depth information. To make such data available to app developers in a device-independent manner, we introduced the Windows.Media.Capture.Frames APIs, a set of extensions to Media Capture that adds frame-by-frame access to RGB, IR, and depth data, as well as sensor correlation to the traditional image and video capture features, all with a consistent, familiar programming model. In addition, this extended model can give UWP apps access to custom streams from a sensor.

In the case of Kinect for Windows, we plan to make use of the custom streams capability to deliver skeleton data from Kinect to UWP apps. This will require a small supplemental Kinect-specific SDK to decode the custom streams on the app side, which we’ll make available later this year.

In order to make use of the new frame-by-frame capabilities of Media Capture, developers need a device capable of delivering rich data streams, as well as an updated driver. A driver update for Kinect will be available later this spring.

By using the proven Media Foundation driver architecture, independent hardware vendors (IHVs) will be able to readily connect their own devices—including RGB, IR, depth, and other custom sensors—to the same Media Foundation infrastructure that now powers Kinect.

The new API, together with the extensible driver model, will empower developers to target Kinect and supported third-party sensors in a hardware-agnostic way to gain better understanding of the world around them and the people in it.

In summary, these are the elements that enable UWP apps to access Kinect:

  • The new Windows.Media.Capture.Frames extensions to Media Capture are available in the Windows 10 SDK preview, build 14332 or newer. These API extensions are device independent. To access IR and depth data, a compatible device and a matching device driver are also required.
  • A matching driver update for Kinect will be available later this spring.
  • To access skeleton data from Kinect, which is delivered through Windows.Media.Capture as a custom stream, a Kinect-specific supplemental SDK is needed, which we plan to release in the second half of 2016.

Join the Windows Insider Program so that you can access Windows 10 build 14332 and its preview SDK. Watch the Channel 9 video for a brief preview of the full capabilities that will be available when the Windows 10 Anniversary SDK and the supplemental Kinect SDK are released to the public, including full skeleton support. The video also tours the code for a simple demo app that demonstrates how to get frames from Kinect or any other RGB camera on your system.

The Kinect for Windows Team

Key links

Comments

  • Anonymous
    May 13, 2016
    This is great! We want the skeletal stream - now - though :)Now I would want a built in "remote kinect" option - and send streams to HoloLens (without having to maintain my own serialization/deserialization layer).
    • Anonymous
      June 01, 2016
      Thanks for your interest. Please participate in our public Kinect for Windows v2 SDK forum, where you can exchange ideas with the Kinect community and Microsoft engineers. You can browse existing topics or ask a new question by clicking the "Ask a question" button on the forum webpage. Access the forum at http://aka.ms/k4wv2forum.
  • Anonymous
    June 04, 2016
    Now if you could give out your Kinect v2 implementation of that new generic API, I guess the community could adapt it to make a Kinect v1 version too (via the Kinect v1.8 SDK or via freenect or whatever they wish), that would be useful for computers with lower capabilities. Plus it would serve as a good basis for other device makers or for the community to support other similar devices too
  • Anonymous
    June 04, 2016
    Would be nice if there was some capability for "Depth Sensor" that a store app could declare as required. That is assuming the new generic API knows if any such device has ever been connected (not necesserily connected at the moment one sees the apps at the store) to the system. Mostly to point users to some page listing such devices (both from MS and from others)
    • Anonymous
      June 15, 2016
      Thanks for your comments! Please check out our public Kinect for Windows v2 SDK forum, where you can exchange ideas like these with the Kinect community, including other devs and Microsoft engineers. You can also browse topics already under discussion or start a new discussion by clicking the Ask a question button.Access the forum at http://aka.ms/k4wv2forum.