With the hand tracking working at a good enough level, the next step was to implement a 'grab gesture' detector. Currently it is not yet very intelligent and only uses the change of the size of the cluster as data source.
But this already enables the user to grad something on the screen and drag it around and then drop it again:
All in all I am surprised how easy it is to do these things with 3D data at hand. The next step will be using two hands to rotate and scale the image.