Mouse Gesture Recognition with Hidden Markov Model.

Mouse‑gesture recognition is a pattern‑recognition problem — gestures are variable but distinctive, and each carries a specific meaning. Because a gesture is a continuous motion in time, Hidden Markov Models (HMMs) are an effective tool for recognition. In this post, I’ll show you how to build a mouse‑motion gesture recognizer using dynamic HMMs, entirely in Java.

Code available @ : https://github.com/gtiwari333/mouse-gesture-recognition-java-hidden-markov-model

Main references

• Tanguay – Mouse Gesture Recognition
• My earlier project: Text‑prompted Remote Speaker Authentication

DEMO VIDEO


How the feature vector is built

Every gesture is described by a set of geometric features computed from the sequence of mouse points:

• Normalised distance of the current point from the centre of gravity (CG) of all points
• Angle of the current point relative to the CG
• Angle of the current point relative to the initial point
• Angle of the current point relative to the end point

To get even more discriminative power, I also added a velocity feature and four additional corner‑based angles:

• Angle of the current point relative to the point with minimum‑x and minimum‑y
• Angle of the current point relative to the point with minimum‑x and maximum‑y
• Angle of the current point relative to the point with maximum‑x and minimum‑y
• Angle of the current point relative to the point with maximum‑x and maximum‑y

System highlights

• Size‑invariant: works regardless of how large you draw the gesture between training and testing
• High accuracy: greater than 95% (detailed analysis still in progress)
• Tested on 30 different gestures — mostly straight‑line shapes and a few curves like ‘S’
• Trained on 1,040 samples
• Angle features are quantised for the discrete HMM
• The system is orientation‑sensitive; rotating a gesture changes its interpretation

Implementation

The core algorithms — k‑means, vector quantisation, codebook generation, Baum‑Welch training, and Viterbi decoding — were taken from my earlier speaker‑authentication project (linked above).

The gesture capture panel uses Java’s MouseMotionListener and MouseListener. This lets me record each (x, y) point along with its timestamp. The timestamps are essential for computing the velocity feature and for replaying a drawing during testing.

Everything is driven from a single GUI: you can recognise gestures, add a new gesture class, supply extra training examples, and retrain both the codebook and the HMM.

HMM parameters

NUM_STATES = 4
NUM_SYMBOLS (codebook size) = 64

Result

I suspected there was a nonzero transition probability to the success state — and indeed there is.

What’s next

I’m extending this work with OpenCV for real‑time hand‑gesture recognition. Check back for the follow‑up.