Body tracking

Article
10/21/2014

This topic discusses how to use the body tracking capabilities of the Kinect Sensor.

BodyFrame

The BodyFrame, accessed from the BodyFrameReader, provides access to the individual bodies through the method GetAndRefreshBodyData. This method takes an IVector<Body> and updates the Body instances with the values from the current BodyFrame. If the IVector is null, then IVector will be filled with a new Body object. The input vector must have the size BodyFrameSource::BodyCount.

This pattern allows for zero-allocation retrieval of the Body object, while still allowing for future additions to the class without breaking.

The following code example shows how to open a sensor, subscribe to the BodyFrameArrived event, and fill the bodies vector with the current body data.

 
void MainPage::InitKinect()
{
  KinectSensor^ sensor = KinectSensor::GetDefault();
  sensor->Open();
  bodyReader = sensor->BodyFrameSource->OpenReader();
  bodyReader->FrameArrived += 
   ref new TypedEventHandler<typename BodyFrameArrivedEventArgs^> (this,                 
                        &MainPage::OnBodyFrameArrived);
  bodies = ref new Platform::Collections::Vector<Body^>(6);
}

void MainPage::OnBodyFrameArrived(BodyFrameReader ^sender, BodyFrameArrivedEventArgs ^eventArgs){
  BodyFrame ^frame = eventArgs->FrameReference->AcquireFrame();
  if (frame != nullptr){
      frame->GetAndRefreshBodyData(bodies);
  }
}

Body data

The bodies tracked by NUI are oriented as if the player were looking into a mirror. This facilitates natural interactions with the game world. For example, to have the player touch an object on the right hand side of the screen, use the right joints:

JointType::HandRight
JointType::HandTipRight
JointType::ThumbRight

Joint normals

There is a normal for each body joint that describes rotation. Rotation is expressed as the vector (in world space) perpendicular to the attached bones in the joint hierarchy. For example, to determine the roll of the right elbow, the immediate parent joint in the hierarchy, the right shoulder, is used to determine the plane of the bone.

Joint hierarchy

Joint hierarchy flows from the center of the body to the extremities, and from the top-most to the bottom-most joints. These connections are described as bones. For example, the bones of the right arm (not counting the thumb) consist of the following connections:

Right hand – Right hand tip
Right wrist – Right hand
Right elbow – Right wrist
Right shoulder – Right elbow

By convention, bones are described as parent–child connections.

What is HandData?

The HandLeftState and HandRightState properties provide information about the state of each of the player's hands. You can use this information to determine if a player is interacting with an object in the title's world.

The states returned are:

Open
Closed
Lasso
NotTracked
Unknown

Using the lasso hand state

Although it is straightforward to think of scenarios in which the open and closed hand states are useful, the lasso state is more complex. The lasso hand state is a closed hand with the middle and index fingers both up. It is like a pointer hand, but with both the index and middle finger. Like the peace sign but the two fingers are together instead of separated. If the hand is large enough and close enough, one finger will likely work as well. The second finger is required to create something large enough in depth to be recognized.

There are no strict rules for using the lasso state. The lasso state may be evaluated in gesture mechanics such as:

Selecting an area by using a circular motion
Activating a region, such as a screen object
Drawing on the screen
Rotating a screen object (for example, rotating a player's avatar during character creation)

There are countless uses for the lasso state. Specific uses will vary by title design and the intended player experience.

Expressions, activities, and appearance

These results are not calculated for the body and exist here for compatibility reasons with Xbox One. If you would like to see expression, activity or appearance information, please see Face tracking for more information.

Engagement

In previous releases, the Body::Enaged property allowed you to determine how engaged a particular body is with the console. This property has been deprecated. Please see Face tracking for how to obtain this data.

Aligning images and body

There are times when you might want to determine the location of points from depth, color, or camera space in another space. For example, if you want to know where in a color image the player’s head is located.

You can use the CoordinateMapper pay-to-play class, which is available from the KinectSensor instance. The following code example shows how to get the color pixel (ColorSpacePoint) represented by the body's head joint.

void MainPage::OnBodyFrameArrived(Platform::Object ^sender, 
                                  BodyFrameArrivedEventArgs ^eventArgs){
{
  BodyFrame ^frame = eventArgs->FrameReference->AcquireFrame();
  if (frame != nullptr)
  {
    frame->GetAndRefreshBodyData(bodies);    

    for each (Body ^body in bodies)
    {
      Joint headJoint = body->Joints->Lookup(JointType::Head);
      CameraSpacePoint headLocation = headJoint.Position;
      TrackingState headTrackingState = headJoint.TrackingState;
        
      CoordinateMapper ^mapper = 
        frame->BodyFrameSource->KinectSensor->CoordinateMapper;
      ColorSpacePoint headPoint = mapper->MapCameraPointToColorSpace(headLocation);
    }
  }
}

Using joint tracking state in filtering

Joints that are inferred are more likely to have temporary spike noises because they are less accurate. Also, the random noise levels are usually higher for inferred joints. In practice, for Kinect games, developers should consider a joint’s tracking state to be valuable information about the quality of the joint data, and apply a more aggressive smoothing filter when a joint’s state is inferred. This can easily be done by checking the joint's tracking state in the filter’s implementation and adaptively updating the filter parameters based on the joint's tracking state. Also, filters that are specifically more powerful in removing spike noise should be applied when a joint's state is inferred.

To learn more about joint filtering techniques, read the topic on Joint Filtering Best Practices.

Share via