Mouse Gesture Recognition with Hidden Markov Model.

Understanding gestures can be posed as a pattern recognition problem. These patterns(gestures) are variable but distinct and have an associated meaning.  Since gesture consists of continuous motion in sequential time, an HMM is an effective recognition tool.

My work is a demonstration of a mouse motion gesture recognition system using Dynamic HMM. It is developed in Java.

Code  Available @ :

Main References were : 

Details :
The gesture features is composed as
  •  Normalized Distance of current point relative to Centre of Gravity of all points 
  •  Angle of current point with respect to  CG
  •  Angle of current point with respect to Initial Pt
  •  Angle of current point with respect to End Pt

To Maximize features and hence performance of system, i also used  velocity feature .Other features included are :
  •  Angle of current point with respect to Min-x and Min-y point 
  •  Angle of current point with respect to Min-x and Max-y point 
  •  Angle of current point with respect to Max-x and Min-y point 
  •  Angle of current point with respect to Max-x and Max-y point 
Features of System :
  •  Invariant with  size of gesture drawing between training and testing
  •  High performance : Greater than 95%. (Detail Analysis is remaining)
  •  Tested with 30 different gestures (mostly composed of straight lines and few curved lines like S)
  •  Trained with 1040 Training data. 
  •  Quantization of angles while composing features
  •  Orientation sensitive
K-means, Vector Quantization, CodeBook Generation, Baum-Welch training, Viterbi Decoding : Algorithm codes were taken from my earlier project : Text-prompted Remote Speaker Authentication.

Gesture Feature Capture Panel coded using  MouseMotionListener and MouseListener, which enables capturing of drawing points with time data. The time information associated with each pixel is required during calculation of velocity feature and replaying the captured feature.

Verify/Recognize, Adding new Gesture and/or adding training data, Training the CodeBook and HMM can be done from main GUI.

Choice of  HMM constants
  • NUM_STATES (number of state in the model) = 4 
  • NUM_SYMBOLS (size of CodeBook) =64
I thought there was a nonzero transition probability to the success state. And I found it to be true.

Next step in this project :I am planning to enhance it by integrating with OpenCV for Hand Gesture Recognition.


  1. What a cool project.
    I want to do something with this theme (mouse gesture recognition with HMM ) for my study.

    What would you advise me ? How deep have I step into the actual Hidden Markov Model theory and its mathematics ?

  2. Hi,
    Good job, really interesting.
    Do you have a PDF report that explains how you applied HMM to analyse the mouse movement ,
    Thank you

    1. I haven't prepared any report for this.
      But I have briefly summarized the Feature Extraction part in my blog above. You can easily understand the implementation my reading the code as well.

      Once you extracted the features, the rest of logic to implement HMM is same. You can read any paper on HMM.

      You can find the good articles on HMM here :

  3. Hey there, as a beginner I'm really looking on what is the exact use of codebook. I tried to run the application without generating codebook and it worked. Could you please elaborate on the same?


Your Comment and Question will help to make this blog better...