Vision-based hand gesture recognition relies on the extraction of features describing the hand, and the appropriate set of features is usually selected in an empirical manner. We propose in this article a systematic selection of the best features to be considered. An iterative sequential forward feature selection (SFS) approach is proposed to combine the features with the highest recognition rate considering the Gaussian Mixture Modelling within the Expectation Maximization algorithm as classification technique. This approach has been tested with two different illustrative databases. The first one is related to human robot physical interaction and the hand postures considered correspond to key postures the human partner performs just before acquiring an object from the robot. The second database corresponds to the representation of the 10 first numbers of the American Sign Language. In both cases, the recognition rate obtained, measured through the F1 score metrics, is satisfactory (over 0,97), and demonstrates that the proposed technique could be applied to a very large field of applications.