Monday, January 30, 2012

Efficient model-based 3D tracking of hand articulations using Kinect

Now this is really, really cool! It's what I've been thinking about for some time now, but haven't started to work on because it looks like a ton of work.

Instead of just trying to detect the fingertips, this approach uses a 3D model of the hand and then tries to orient and bend it to fit with the data from the Kinect. Because it's a complete model of the hand you not only find the positions for the fingertips, but also all other joints. This makes my work more than obsolete ;-)

  
Reference:
I. Oikonomidis, N. Kyriazis and A.A. Argyros, "Efficient model-based 3D tracking of hand articulations using Kinect", to appear in Proceedings of the 22nd British Machine Vision Conference, BMVC 2011, University of Dundee, UK, Aug. 29-Sep. 1, 2011.

Here's the link to the project site (more videos)

19 comments:

  1. Cool, but it doesn't make your work obsolete.
    A robust and good working API that works with the Microsoft SDK is very much needed!

    ReplyDelete
  2. So ... is that the end of your Kinect career? :-) ... since the lumia is waiting for fresh bits anyways ...

    ReplyDelete
  3. i saw this video be4, it is great
    while i dont know what is the algorithm behind this
    there are 2 things that is quite significant
    it uses skin color segmentation
    the finger location is some how predicted by the algorithm
    based on this video http://www.youtube.com/watch?v=N3ffgj1bBGw

    ReplyDelete
  4. It is amazing!
    Are you going to make smth similiar?
    It's definitely worth trying.
    Is it open-source?

    ReplyDelete
    Replies
    1. I have contacted the author of the project.
      He answered:
      "Our intention is to make libraries available to the community for research purposes. Unfortunately, due to many other commitments, this takes time. As soon as we have it, I can let you know."

      Delete
    2. Dear all,

      An executable demo version of our Kinect 3D hand tracking software is now available for download at the following web page:

      http://cvrlcode.ics.forth.gr/handtracking/

      Best regards,

      Iason - Nikolaos - Antonis

      Delete
  5. @Afra: Thanks! I think the model based approach is the future. Currently they have to run it on a GPU and processing only happens at 12 Hz, so there's some time left before I have to stop :-)

    @Jonas: I hope not :-) Sooner or later Microsoft and/or PrimeSense will add finger tracking to their frameworks, until then I can keep Candescent NUI running. And I can also start creating "end user" stuff.

    @tcboy: They give a hint: Particle Swarm Optimization

    @Eduard: I already thought about creating a 3D model and then combine this with my current finger detection, so I would put the fingers where they are detected. This would be somewhere between the two approaches... I'll have to see if I can do it with the time I'll find (unfortunately I don't have a research budget...) If I'd come up with something that works as well as shown in their video I'd have to sell it ;-)

    Thanks for asking the author!

    ReplyDelete
  6. http://www.microsoft.com/en-us/kinectforwindows/news/faq.aspx
    Kinect for windows is launched
    the sdk will still works for old kinect too
    BUT not sure about the near mode

    ReplyDelete
  7. Hi tcboy

    As far as I know the near mode is only available with the new hardware. I'm installing the new SDK right now on my laptop (got OpenNI on my main PC).

    - Stefan

    ReplyDelete
  8. Hi Stefan.
    Thank you very much for your project.
    I point out the following link for code migration (MS Kinect SDK v1):
    http://robrelyea.wordpress.com/2012/02/01/k4w-code-migration-from-beta2-to-v1-0-managed/
    What efficiencies (in terms of fps, Hz) can you achieve for fingertips with xtion pro?
    It would be interesting to develop a similar approach to the hand combined with your project:
    http://www.vision.ee.ethz.ch/~gfanelli/head_pose/head_forest.html
    Where can I find the source code for the Video Manipulation Interface that I can test?
    I'd like to give some suggestions to improve the project after having had time to analyze the code.
    Best wishes,
    Roberto

    ReplyDelete
  9. it has been a long time since u last updated ur blog =(

    ReplyDelete
  10. Hi tcboy

    I've been snow shoe trekking for a few days and now I've got a cold...

    ReplyDelete
  11. it must be fun! have u seen aurora?
    anyway take good care

    ReplyDelete
  12. Unfortunately no, you don't usually see Aurorae in Switzerland...

    I'm fit enough to sit on the computer again. I should redesign my website.. candescent.ch

    ReplyDelete
  13. I just submitted 2x 50 pages report to my professor (about computer networking)
    now I have some free time before next semester start
    I shall concentrate on some kinect thingy!

    ReplyDelete
  14. Hi Tcboy

    Cool! Too bad the Kinect didn't exist when I was at university.

    I'm overhauling my website, I hope to have the new version online after the weekend (www.candescent.ch)

    - Stefan

    ReplyDelete
  15. i m doing master by research now, just finished 1st semester
    to be honest, i am in the wrong field now (computer networking)
    while my interested field is actually HCI and VR
    hope i could switch professor and lab, but it is unlikely to happen =(

    anyway, looking forward for ur new website interface

    ReplyDelete
  16. Hi Stefan,

    Your work is amazing. I am in my final year at university and doing some similar project like yours. May I use some of your idea? I will surely cite you and your site. Btw, the FORTH library is now available. I've just tried the testing example (.exe file), it works. However, my PC's GPU is so weak ( I guess its price is around 80 quids, while the minimum one they say in their site is at least >300 quids to obtain a fps of 20).

    tnq177

    ReplyDelete
  17. Hi Toan

    Thanks! You're welcome to use my project, as long as you add it in the credits.

    Unfortunately I can't try it, because I don't have an NVIDIA graphics card.

    - Stefan

    ReplyDelete