Sunday, August 14, 2011

KinectMultiTouchDevice for WPF 4

Today I've published the KinectMultiTouchDevice class on CodePlex. It allows you to leverage the multi-touch support added by Microsoft to WPF 4 and combining it with Kinect! (Here's an introduction blog post that covers much of multi-touch in WPF 4).

Multi-touch with Kinect and WPF 4

The update comes with two simple WPF samples in a new Candescent NUI WPF Samples project.
It also works with the WPF samples of Microsoft Surface 2.0 SDK, but you'll have to add the KinectTouchDevice for each demo project manually.

Screen capture of the multi-touch sample project

All fingers of both hands can be used simultaneously. Though it rarely makes sense to use more than 1 or two per hand.

Please note: I've only published the source code, I'll add a binary release soon. The sample only runs with OpenNI for the moment.

Technical details
The class CCT.NUI.Touch.KinectMultiTouchDevice can be found in the CCT.NUI.HandTracking.dll.

In a WPF 4 window it can be initialized like this (in the loaded event, you must should not add this code in the constructor).

 this.factory = new OpenNIDataSourceFactory("config.xml");  
 var handDataSource = new HandDataSource(this.factory.CreateClusterDataSource());  
 this.multiTouchDevice = new KinectMultiTouchDevice(handDataSource, this);  
 handDataSource.Start();

The device identifiers are created the following way: The ID is always a 2 digit decimal number where the first number is the hand ID and the second number the finger ID. Finger IDs are assigned in order of appearance and do not identify a specific finger (for example index).

There are some glitches here and there, I'll keep working on it. For now I'm looking forward to see what you'll do with it!

Other changes in changeset 9161
  • New mapping algorithm
    Hands and fingers are mapped between two frames and keep the same ID until they disappear. IDs are reused! That means if Hand 1 is removed from view the next hand that appears will get ID 1 again. The same is true for the fingers.
  • Fingers and Hands implement the ILocatable interface
  • HandData.Id is now int instead of guid
  • Finger points get IDs. They are assigned in order of appearance, so you can't assume that 1 is always the thumb. Also the IDs are reused.
  • Option to show depth view in Candescent StartMenu
  • Exit button in Candescent Start Menu

15 comments:

  1. i cant get it working

    ReplyDelete
  2. Hi Anonymous

    I'd need some more information to be able to help you.

    Regards,
    Stefan

    ReplyDelete
  3. i get this error Unable to find an entry point named 'xnInitFromXmlFileEx' in DLL 'OpenNI'. then i changed the OpenNI version and it worked.

    ReplyDelete
  4. Good to hear!

    For those who have the same problem: Please use the newest unstable version v1.3.2.3 dated 19th July from OpenNI.org

    ReplyDelete
  5. amazing, I cannot wait to test it.

    ReplyDelete
  6. Hy, I have a problem with testing your
    applications. I now here is off topic but anyway, when I try to build the solution, it gives me this error:


    Error 1 Error generating Win32 resource: Error reading icon 'C:\PATH\CCT.NUI.Samples\organizer.ico' -- The system cannot find the file specified.

    I tried to find that organiser.ico on codeplex but wasn't available for download. Can you help me please.. Thanks

    ReplyDelete
  7. Hi Anonymous

    You're right, the icon was missing. Please try changeset 9489, it should work now.

    Regards,
    Stefan

    ReplyDelete
  8. Hi,

    You mentioned above that you could add Kinect touch input to the Surface Demo examples by manually adding a KinectTouchDevice to the project. I tried using your code above but was getting errors. this.factory and this.multiTouchDevice were unrecognized despite having referenced CCT.NUI.HandTracking, CCT.NUI.Core, CCT.NUI.Touch, and OpenNI.

    Please help!

    ReplyDelete
  9. Hi Anonymous

    You have to add the following two fields to the class where you added the other code:

    private IDataSourceFactory factory;
    private KinectMultiTouchDevice multiTouchDevice;

    Regards,
    Stefan

    ReplyDelete
  10. Thank you! It works now!

    ReplyDelete
  11. HI,
    I'm trying to use your project(candescentnui-9161)
    But the VS 2010 Express tell me that I miss the Microsoft.VisualStudio.TestTools.UnitTesting.
    How can I solve this problem? Thanks.

    ReplyDelete
  12. Hi Anonymous

    VS Express does not support Microsofts Testing Framework.

    You can remove the test project from the solution, the rest should work.

    Regards,
    Stefan

    ReplyDelete
  13. Hi Stefan,
    I'm studing your application and I have a question: it's possible using the SDK skeleton tracking for having hand data instead of using OpenNi hand tracking? And do you know if it's possibile use the skeleton joints as touch controls? Thanks in advance, Deby.

    ReplyDelete
  14. Hi Deby

    Out of the box no, but it would not be too hard to modify the code to use an area around the hand joint for finger detection.

    But the thing is that skeleton tracking works best if you're 2 meters away from the device while my hand detection only works up to 1 meter (maybe a little more).

    - Stefan

    ReplyDelete