Raw pixels are the wrong feature for a hand.
Classifying gestures straight from webcam pixels forces a model to learn lighting, skin tone, backgrounds, and hand position all at once — most of which have nothing to do with the gesture. The signal that actually matters is the geometry of the hand: where the fingers are relative to each other.
MediaPipe hands that geometry over directly as 21 landmark points, so the real design question becomes how to turn those coordinates into a feature a small classifier can learn from reliably.
Recognition shouldn't care where your hand is, only what it's doing.
What the recogniser had to do.
Run in real time on a webcam.
Landmark detection and classification per frame, fast enough that the on-screen label and audio keep up with a moving hand.
Speak, don't just show.
Non-blocking text-to-speech via gTTS announces the gesture, so the tool works as accessible feedback and not only a visual overlay.
Be trainable by the user.
Record, train, recognise — a full CLI so anyone can capture their own gestures and fit a custom model.
Detect, normalise, classify, speak.
Each frame runs through MediaPipe for landmarks, a normalisation step to make the pose position- and scale-invariant, a TensorFlow classifier for the label, and a threaded audio manager for spoken feedback.
The codebase is organised into a proper package — tracking, services (audio, smoothing, performance), and UI — rather than one long script, with prediction smoothing to stop the label flickering between frames.
The model learned position, not gesture.
The original recorder stored raw pixel coordinates, so an early model quietly learned "thumbs up in the centre of the frame." Move your hand to the corner and recognition fell apart — it had memorised where the training hands sat, not the shape they made.
The fix was wrist-relative, scale-invariant normalisation: re-centre every landmark set on the wrist and normalise its size, so the same gesture looks identical anywhere in the frame at any distance. It's the single change that turned a demo into something that actually generalises — and any model trained before it needs a retrain.
A tool, not a script.
The recogniser ships as a CLI with distinct modes — recognise, record, train, settings — plus a settings dialog and environment-variable config for detection sensitivity and camera index. That's what lets someone extend it to their own gestures instead of forking and editing constants, and it's why the normalisation lives in shared code both training and inference call.
Three things, next time.
Support two-handed gestures.
The pipeline handles one hand; many useful signs need both, which means rethinking the feature vector.
Add temporal gestures.
Static poses miss motion-based signs; a small sequence model over frames would unlock waves, swipes, and dynamic signs.
Cache speech for common labels.
gTTS needs the network per phrase; pre-rendering the fixed gesture names would make audio instant and offline-capable.