Mouse Gesture Recognition with Hidden Markov Model.

Understanding gestures can be posed as a pattern recognition problem. These patterns(gestures) are variable but distinct and have an associated meaning.  Since gesture consists of continuous motion in sequential time, an HMM is an effective recognition tool.

My work is a demonstration of a mouse motion gesture recognition system using Dynamic HMM. It is developed in Java.

Code  Available @ : https://github.com/gtiwari333/mouse-gesture-recognition-java-hidden-markov-model

Main References were : 

•  http://robotics.stanford.edu/~tanguay/docs/meng.pdf 
•  My Final Year Project 

DEMO VIDEO 


Details :
The gesture features is composed as

•  Normalized Distance of current point relative to Centre of Gravity of all points 
•  Angle of current point with respect to  CG
•  Angle of current point with respect to Initial Pt
•  Angle of current point with respect to End Pt

To Maximize features and hence performance of system, i also used  velocity feature .Other features included are :

•  Angle of current point with respect to Min-x and Min-y point 
•  Angle of current point with respect to Min-x and Max-y point 
•  Angle of current point with respect to Max-x and Min-y point 
•  Angle of current point with respect to Max-x and Max-y point 

Features of System :

•  Invariant with  size of gesture drawing between training and testing
•  High performance : Greater than 95%. (Detail Analysis is remaining)
•  Tested with 30 different gestures (mostly composed of straight lines and few curved lines like S)
•  Trained with 1040 Training data. 
•  Quantization of angles while composing features
•  Orientation sensitive

Implementation
K-means, Vector Quantization, CodeBook Generation, Baum-Welch training, Viterbi Decoding : Algorithm codes were taken from my earlier project : Text-prompted Remote Speaker Authentication.

Gesture Feature Capture Panel coded using  MouseMotionListener and MouseListener, which enables capturing of drawing points with time data. The time information associated with each pixel is required during calculation of velocity feature and replaying the captured feature.

Verify/Recognize, Adding new Gesture and/or adding training data, Training the CodeBook and HMM can be done from main GUI.

Choice of  HMM constants

• NUM_STATES (number of state in the model) = 4 
• NUM_SYMBOLS (size of CodeBook) =64

Result
I thought there was a nonzero transition probability to the success state. And I found it to be true.

Next step in this project :I am planning to enhance it by integrating with OpenCV for Hand Gesture Recognition.