How to track head position

I want to do something similar to what Johhny Lee did in his Wii head tracking http://www.youtube.com/watch?v=Jd3-eiid-Uw&feature=player_embedded

But I wa开发者_开发技巧nt to use the Kinect. Since Microsoft's sdk exposes the skeletal joints, I had hoped I might be able to just use that to get the head position. The problem is that I want to do this with my desktop computer and its monitor. If I put the Kinect sensor right next to my monitor and sit at the desk. pretty much just my head and neck are visible to the sensor, so the skeletal tracking doesnt pickup on my head position.

Is anyone familiar with a head tracking project using the Kinect? Preferably in C#

I think for this application you are not able to use skeleton tracking provided by any framework such as Microsoft's SDK or OpenNI.

I would propose to segment the head of the user by applying a depth threshold to the raw depth data. This should result in background reduction. I think there are already existing methods which do this.

As second step you woud like to have something like an axis inside the segmented user. The simplest approach would be to use opencv fitEllipse. The major axis of the returned ellipse combined with the depth information gives you this axis.

This approach only works when most segmented points belong to the head of the user. If you are further away you have to think about a method to segment only the head. The ellipse fitting should always work.

There are limitations on the offical Kinect for Windows SDK, which fall in-lines with the guidance provided for the XBox and XDK in that you need to be 1.2m to 3.5m away from the sensor to be able to use the Kinect sensor. This limitation is actually reduced in alternate SDKs such as the OpenNI/NITE libraries that would allow you to detect skeletons/objects closer to the sensor.

The problem you will also have with the skeltal input is it will only detect the position of the head in proportion to the skeleton, but not if you rotate your head side to side. To achieve this you would not to use the raw depth streams and some smarts around object recognition which is somewhat more complicated.

In the past I've used this commercial .NET API which uses a webcam to track head movements, and achieves what you are after: http://luxand.com/facesdk/index2.php

You don't need the kinect to track your head position. You can do the same thing with a regular camera and openCV by utilizing face-tracking.

Simple example showed here: http://vimeo.com/19464641

In the video I am using openCV to track my face (which you can barely see in the corner, but the red dot indicates my face position).

Check out Channel 9s tutorials on this kind of subject. You would go to the Skeletal Fundamentals Video. But here's some code if you want to save the time.
XAML

<Window x:Class="SkeletalTracking.MainWindow"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="MainWindow" Height="600" Width="800" Loaded="Window_Loaded" 
    xmlns:my="clr-namespace:Microsoft.Samples.Kinect.WpfViewers;assembly=Microsoft.Samples.Kinect.WpfViewers" 
    Closing="Window_Closing" WindowState="Maximized">       
<Canvas Name="MainCanvas">
    <my:KinectColorViewer Canvas.Left="0" Canvas.Top="0" Width="640" Height="480" Name="kinectColorViewer1" 
                          Kinect="{Binding ElementName=kinectSensorChooser1, Path=Kinect}" />
    <my:KinectSensorChooser Canvas.Left="250" Canvas.Top="380" Name="kinectSensorChooser1" Width="328" />
    <Image Canvas.Left="66" Canvas.Top="90" Height="87" Name="headImage" Stretch="Fill" Width="84" Source="/SkeletalTracking;component/c4f-color.png" />
</Canvas>

Internal Code

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using Microsoft.Kinect;
using Coding4Fun.Kinect.Wpf; 

namespace SkeletalTracking
{
/// <summary>
/// Interaction logic for MainWindow.xaml
/// </summary>
public partial class MainWindow : Window
{
    public MainWindow()
    {
        InitializeComponent();
    }

    bool closing = false;
    const int skeletonCount = 6; 
    Skeleton[] allSkeletons = new Skeleton[skeletonCount];

    private void Window_Loaded(object sender, RoutedEventArgs e)
    {
        kinectSensorChooser1.KinectSensorChanged += new DependencyPropertyChangedEventHandler(kinectSensorChooser1_KinectSensorChanged);

    }

    void kinectSensorChooser1_KinectSensorChanged(object sender, DependencyPropertyChangedEventArgs e)
    {
        KinectSensor old = (KinectSensor)e.OldValue;

        StopKinect(old);

        KinectSensor sensor = (KinectSensor)e.NewValue;

        if (sensor == null)
        {
            return;
        }




        var parameters = new TransformSmoothParameters
        {
            Smoothing = 0.3f,
            Correction = 0.0f,
            Prediction = 0.0f,
            JitterRadius = 1.0f,
            MaxDeviationRadius = 0.5f
        };
        //sensor.SkeletonStream.Enable(parameters);

        sensor.SkeletonStream.Enable();

        sensor.AllFramesReady += new EventHandler<AllFramesReadyEventArgs>(sensor_AllFramesReady);
        sensor.DepthStream.Enable(DepthImageFormat.Resolution640x480Fps30); 
        sensor.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);

        try
        {
            sensor.Start();
        }
        catch (System.IO.IOException)
        {
            kinectSensorChooser1.AppConflictOccurred();
        }
    }

    void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e)
    {
        if (closing)
        {
            return;
        }

        //Get a skeleton
        Skeleton first =  GetFirstSkeleton(e);

        if (first == null)
        {
            return; 
        }



        //set scaled position
        ScalePosition(headImage, first.Joints[JointType.Head]);
        //ScalePosition(leftEllipse, first.Joints[JointType.HandLeft]);
        //ScalePosition(rightEllipse, first.Joints[JointType.HandRight]);

        GetCameraPoint(first, e); 

    }

    void GetCameraPoint(Skeleton first, AllFramesReadyEventArgs e)
    {

        using (DepthImageFrame depth = e.OpenDepthImageFrame())
        {
            if (depth == null ||
                kinectSensorChooser1.Kinect == null)
            {
                return;
            }


            //Map a joint location to a point on the depth map
            //head
            DepthImagePoint headDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.Head].Position);
            //left hand
            DepthImagePoint leftDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandLeft].Position);
            //right hand
            DepthImagePoint rightDepthPoint =
                depth.MapFromSkeletonPoint(first.Joints[JointType.HandRight].Position);


            //Map a depth point to a point on the color image
            //head
            ColorImagePoint headColorPoint =
                depth.MapToColorImagePoint(headDepthPoint.X, headDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //left hand
            ColorImagePoint leftColorPoint =
                depth.MapToColorImagePoint(leftDepthPoint.X, leftDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);
            //right hand
            ColorImagePoint rightColorPoint =
                depth.MapToColorImagePoint(rightDepthPoint.X, rightDepthPoint.Y,
                ColorImageFormat.RgbResolution640x480Fps30);


            //Set location
            CameraPosition(headImage, headColorPoint);
            //CameraPosition(leftEllipse, leftColorPoint);
            //CameraPosition(rightEllipse, rightColorPoint);
        }        
    }


    Skeleton GetFirstSkeleton(AllFramesReadyEventArgs e)
    {
        using (SkeletonFrame skeletonFrameData = e.OpenSkeletonFrame())
        {
            if (skeletonFrameData == null)
            {
                return null; 
            }


            skeletonFrameData.CopySkeletonDataTo(allSkeletons);

            //get the first tracked skeleton
            Skeleton first = (from s in allSkeletons
                                     where s.TrackingState == SkeletonTrackingState.Tracked
                                     select s).FirstOrDefault();

            return first;

        }
    }

    private void StopKinect(KinectSensor sensor)
    {
        if (sensor != null)
        {
            if (sensor.IsRunning)
            {
                //stop sensor 
                sensor.Stop();

                //stop audio if not null
                if (sensor.AudioSource != null)
                {
                    sensor.AudioSource.Stop();
                }


            }
        }
    }

    private void CameraPosition(FrameworkElement element, ColorImagePoint point)
    {
        //Divide by 2 for width and height so point is right in the middle 
        // instead of in top/left corner
        Canvas.SetLeft(element, point.X - element.Width / 2);
        Canvas.SetTop(element, point.Y - element.Height / 2);

    }

    private void ScalePosition(FrameworkElement element, Joint joint)
    {
        //convert the value to X/Y
        //Joint scaledJoint = joint.ScaleTo(1280, 720); 

        //convert & scale (.3 = means 1/3 of joint distance)
        //Joint scaledJoint = joint.ScaleTo(1280, 720, .3f, .3f);

        Canvas.SetLeft(element, scaledJoint.Position.X);
        Canvas.SetTop(element, scaledJoint.Position.Y); 

    }


    private void Window_Closing(object sender, System.ComponentModel.CancelEventArgs e)
    {
        closing = true; 
        StopKinect(kinectSensorChooser1.Kinect); 
    }



   }
}

I personally would recommend watching the videos because they explain everything. Good luck on your project!

I suggest to use: Aforge.net together with Microsoft XNA Framework or just Aforge.net alone. You will need to do a bit of development yourself though. I am also working on similar thing using C#. I think you won't be able to find a complete out-of-the-box example. No body did that yet. (Correct me if I am wrong).