Since then I was able to refine it considerably. The key was to use the hand's contour and then combine this information with the points in the convex hull. Here is a video that showcases the current algorithm:
1. For each point in the hull (candidates for fingertips), find the nearest point in the contour curve. Let's call this set C.
2. For each point c in C, take the two points (p1, p2) in the two different directions along the contour that are in a given distance to c (the ideal distance has to be found experimentally and depends on the handshape's size).
3. If these three points are aligned, then it's not a fingertip point. To find out if they are aligned, find the center of p1 and p2 and calculate the distance to c. If this distance is bigger than a certain value (to be found experimentally), the points are not on a line and the cantidate point c is a fingertip point.
c is a fingertip, while c' is not.
This version also works when the fingers are pointing down:
Next I will try to find skin areas with the RGB camera, then use these areas to find hand shapes. And the current code base needs some heavy refactoring.