Saturday, February 18, 2012

Microsofts Finger Tracking

I've only found this today, it's online since three months already. It's a slideshow and audio recording about Microsofts (actually it says "Good Science Studios") progress on finger tracking.

http://www.microsoft.com/download/en/details.aspx?id=27977

They're providing quite some details on how they are doing it.

34 comments:

  1. lol i was browsing the gamefest 2011 slides 2 days ago but didnt notice this
    i found some slides about gesture recognition but it was to deep for me

    ReplyDelete
  2. http://youtu.be/J9nGI23Knhs

    similar to ur work, wondering if it was his own code or taken from your source =x

    ReplyDelete
  3. Hi TcBoy

    Interesting :-) He writes that he was inspired by my video.

    I haven't published the code for the video manipulation yet, so it's definitely his work. Don't know how he's doing the finger tracking.

    - Stefan

    ReplyDelete
  4. Sorry, I want to ask a question.

    If I want to indicate the distance value on image stream video(not

    indicate on depth stream video), how do I do that?

    Can you give me some suggestion?

    Thanks!

    Berry

    ReplyDelete
  5. Hi Berry

    How do you want to render the image stream? With WinForms (Bitmap) or WPF (ImageSource)?

    - Stefan

    ReplyDelete
  6. Actually not render, it is close image segmentation.

    For example, I want to show the RGB image that their depth value are

    smaller than 1m.

    By the way, it is with WinForms.

    But if I can render the image stream, may be I can do the same thing?

    Because I can get the x-y coordinate value, but I'm not sure.

    I'm new to Kinect:(

    Thanks!

    Berry

    ReplyDelete
  7. In this case you need to find the pixels that are in the desired distance in the depth stream first, store their coordinates (x,y) somewhere and then process the video bitmap and filter out the ones that don't match.

    You'll need to tell OpenNI to align depth and video image. I've added a method on the OpenNIDataSourceFactory that does this:
    SetAlternativeViewpointCapability

    ReplyDelete
  8. I see, I will try to trace your code and implement

    it.

    Thanks!

    Berry

    ReplyDelete
  9. http://labs.vectorform.com/2011/06/video-exploring-kinect%E2%80%99s-hand-recognition-capabilities/#more-591

    "To do this I approximated a polygon around the contour, then I did a broad search around the center for the spot that had the maximum distance from all points on the contour. Once I found this spot I performed another search around that area to increase accuracy."

    instead of calculating all the points in contour, maybe this idea can reduce the computational time.

    ReplyDelete
  10. Hi Tcboy

    I'm reducing the points in the contour and also I'm only checking every x-th point inside the contour and closer around the first solution in a second step.

    But I guess the center of the convex hull might be a good starting point. I'll investigate when I find some time.

    - Stefan

    ReplyDelete
  11. Ya I did the same, only checking every 4 to 5-th points
    but the computational time still quite long, i dont know why
    a for loop checking about 1000 pixels takes that long?
    it get worse when the hand become bigger
    it kinda lag my program =(
    so I was thinking how to reduce the computational time there
    just sharing

    ReplyDelete
  12. If you compare every pixel inside the contour with each pixel on the contour this multiplies (for example 1000 x 100 = 100'000 computations). And when the hand becomes bigger there are more pixels inside and on the contour.

    I'll try only searching around the center of the convex hull. This could speed things up quite a bit. And maybe even improve some cases where the center would be outside the contour in my algorithm.

    ReplyDelete
  13. i was using a nice opencv function "cvPointPolygonTest"
    which will return the shortest distance from inner points to nearest contour

    anyway, how do you perform your depth threshold? sorry, was lazy to study on ur codeplex becoz it is too hard for me =(
    i was using multiple histogram but it slows down performance alot

    ReplyDelete
  14. The depth threshold is directly using the distance values as they are returned by the depth stream (millimeters) without histogram. That's only used when rendering the image.

    ReplyDelete
  15. Sorry, I faced a problem and I don't know how to solve it. I use an two-dimension array to storage the pixels that are in the desired distance in the depth stream first, but I don't know how to send it to video stream and synchronously filter out that ones don't match , because the event parameter has its format.
    I can't change it!
    Can you give me some advise?
    Thanks!
    Berry

    ReplyDelete
  16. http://www.engadget.com/2012/02/28/microsoft-techforum-unveils-three-research-projects-video/

    some cool videos, especially the first 1

    ReplyDelete
  17. @berry I'm not sure what the problem is exactly. You need to synchronize the depth and video stream, then first process the depth map and then the rgb image.

    @tcboy I'd like to work on such projects :-)

    ReplyDelete
  18. http://www.softkinetic.com/Solutions/DepthSensecameras.aspx

    15cm min depth o.O
    I am thinking to mount it around my chest lol

    ReplyDelete
  19. http://www.theverge.com/2012/1/15/2705764/softkinetic-depth-sensor-311-320-alpha-finger-controls

    ReplyDelete
  20. Hi Tcboy

    Interesting! They're around for quite some time. I didn't know them before. Their device costs 500$.

    - Stefan

    ReplyDelete
  21. 500usd is too much...
    maybe nyko zoom is the better alternative
    some ppl use kinect with nyko zoom can reach around 32cm
    xtion pro live with nyko zoom can reach around 27cm

    source:
    http://groups.google.com/group/openni-dev/browse_thread/thread/a1baa822c8ea375c

    ReplyDelete
  22. I still wasn't able to buy a K4W...

    ReplyDelete
  23. why do u need to get k4w?
    for the licensing?

    ReplyDelete
  24. the near mode of k4w is only capable of 40cm
    while old kinect w nyko zoom probably can reach 32cm
    it is a much cheaper alternative
    i may buy nyko soon, i will feedback to u then =)

    ReplyDelete
  25. I want to support it in Candescent NUI.. and to test it, I need the hardware...

    Ok, feedback is always good :-)

    ReplyDelete
  26. Have you seen the kinect challenge?
    http://www.kaggle.com/c/GestureChallenge
    Will be fun to participate

    ReplyDelete
  27. Hi Eduard

    Thanks for the link! I hadn't seen this before. Looks interesting, but maybe it's a bit late to start taking part now, it started 3 month ago.

    - Stefan

    ReplyDelete
    Replies
    1. What really matters is how many days left. We have more than a month.

      May be we could collaborate with you and readers of this blog.

      Delete
    2. One month isn't really much... but I like the idea. I'll look into it in more details later today.

      Delete
  28. Hi Stefan
    I want to know how to recognize the "Click" gesture of finger.

    ReplyDelete
  29. Hi Jingjing

    That depends on what you define as "click" gesture. You could define it as moving one finger closer to the kinect while the other fingers stay in the same plane.

    - Stefan

    ReplyDelete
  30. http://www.opennichallenge.com/opennichallenge.php

    good luck Stefan!

    ReplyDelete
  31. Hi TcBoy

    Thanks :-)
    I might make a photo tool. Don't know about the prize though.. Not sure if I could even go.

    - Stefan

    ReplyDelete