Until now I was using the center of the cluster as center of the hand (the dark blue dot). But this is not ideal, because if you open and close your hand, the center moves up and down. The same happens if you rotate the hand or if more of the arm becomes visible.
The solution is to find the center of the palm, which is quite stable during rotating, opening and closing the hand (the light green dot). This is done by finding the biggest circle inside the hands contour. The center of this circle is in most cases the center of the palm.
This circle can be found by identifying the point inside the contour that maximizes the distance to the points in the contour line. The line through the center of the cluster and the center of the palm could also be used as hand orientation indicator.
Blue point: Center of the cluster
Green point: Center of the palm
Red points: Finger tips
White line: Convex hull
Yellow line: Contour
Red line: Distance from the palm point to the contour
Green circle: Biggest circle that is completely within the contour