Thursday, July 7, 2011

Hand Detection Performance

I've worked on the performance of the hand detection, here is a summary:

When I started the optimizations the detection of a hand took around 22 - 23ms (which is a lot if you only have time for 33ms per frame). The distribution was like this:

Finding the contour
Used to take 12ms per frame (so this accounts for half of the time used). I've reduced this to 9ms and think there is still potential.

Center of the palm
Used to take 7ms per frame. Now it takes 3.5ms and is also more accurate.

Mapping the hands in the new frame to the hands in the old frame
3.5ms per frame, I didn't try to optimize this yet.

Finding the convex hull
< 1 ms per frame, I didn't try to optimize this yet.

Detecting the fingers
< 1 ms per frame, I didn't try to optimize this yet.

With the improvements I've got the total time down to 16ms (around 30% less).


I've introduced TPL (Task Parallel Library) to take advantage of multi core CPUs. Example from PalmFinder.cs:

Parallel.For(0, candidates.Count, (index) =>
{
    distances[index] = FindMaxDistance(contour, candidates[index]);
});

This will start multiple threads (the actual number depends on the number of cores that are available) to calculate the distances for the candidate points.

I've also made some variables that were used in loops method scoped instead of class scoped, this also increased speed.


This was measured on my Intel Core i7 860  (this is a quad core CPU). Speed improvements on two or single core CPUs might be smaller.

Edit: The new version is on CodePlex: 8454

12 comments:

  1. seems a little slower but more consistent on my Dual code AMD (KinectSDK) - wil revert to OpenNI drivers soon

    ReplyDelete
  2. here are some links where they seem to be able to get both SDKs to work with each other. i'll take a thorough look too. http://groups.google.com/group/openni-dev/browse_thread/thread/22cd12bc8da97e9e

    ReplyDelete
  3. Hi EddHead

    From what I've read (just a quick scan), they use the SDK depth data and feed it to the NITE implementation. So They are able to use both skeleton tracking algorithms etc. But it won't remove any depth limitations from the depth stream.

    Regards,
    Stefan

    ReplyDelete
  4. Awesome!
    I'm more interested in your project. I have one question to ask you. Why don't you expand depth map? I don't think its possible because of clustering working, is that right?

    ReplyDelete
  5. Hi Anonymous

    Thanks! There are two reasons:

    - You are right. The clustering will take everything that's within the depth range, it can't decide if it is a hand or something else.

    - The resolution (640x480) makes it hard to seperate fingers when the distance is 1m or more.

    Regards,
    Stefan

    ReplyDelete
  6. Great work! I was just wondering how you measured the performance. Are you using a profiling tool and if so which one?

    Regards, Vicky

    ReplyDelete
  7. Hi Vicky

    Thanks for the comment!

    Unfortunately it seems that there is no usable free or even open source performance profiler around for .NET (please correct me if I'm wrong, I'd like to know about it!)

    For example the Red Gate ANTS Profiler costs 395$ for a single license of the standard version, and since I'm not making any money from this project (apart from the 8 clicks on the ads I've had so far ;-) ) that's too expensive.

    So I've just added some stop watch code myself.

    Regards,
    Stefan

    ReplyDelete
  8. Hi Stefan,

    I have tried using stop watch code too but I find the timings are quite variable when used inside NUnit tests. Do you have the same problem?

    I have been looking for a free profiler myself and I found that there is a 10 day free trial of DotTrace:

    http://www.jetbrains.com/profiler/index.html?topDT

    They also have a free Open Source licence for Open Source projects that have been running for at least 3 months, see here for more details:

    http://www.jetbrains.com/profiler/buy/buy.jsp#openSource

    I haven’t tried it yet but I am a big fan of one of their other .NET products Resharper.

    Thanks,
    Vicky

    ReplyDelete
  9. Hi Vicky

    In my tests I'm running the algorithm over the same frame around 200 times, so it should give a good average value, it still varies around 1/10th (+/- 0.5ms per 10ms for example) but I think that's ok for the moment.

    In fact I've created the CodePlex project around 3 months ago, so I could apply for a free version. I guess I'll give it a try.

    You don't happen to work for Jetbrains, do you? ;-)

    Thanks too!
    Stefan

    ReplyDelete
  10. Vicky,

    I've applied for an open source license of dotTrace. Now I have to wait and see if they accept the request.

    ReplyDelete
  11. Follow up:
    Today I got a dotTrace open source license key :-)

    ReplyDelete