Building Hand Fighter: from a webcam experiment to online matches
Hand Fighter started with a single question: if a webcam can see a hand, can a hand be a controller? This is the story of how that question became a fighting game with six characters, three stages, an arcade ladder and online rooms, and the decisions that shaped it along the way.
Seeing a hand
Modern browsers can run a hand-tracking model locally. Point the camera at a hand and, about thirty times a second, you get the positions of 21 points: the wrist, and four joints on each finger. That is the raw material. None of it is sent anywhere; the model runs on the device's own processor, which is why the game can promise that no video is ever uploaded.
The first prototype simply drew those 21 points on screen. It was mesmerising and completely useless as a game. The interesting work was turning points into intentions.
From points to gestures
A fighting game needs a handful of clear inputs: move, jump, punch, kick, block. We mapped them to hand shapes that are easy to make and easy to tell apart:
- Open palm tilted left or right moves the fighter. The game measures the lean of the hand relative to the wrist.
- Raising the open hand high in the frame jumps.
- A fist punches. All fingers curled.
- Two fingers kick. Index and middle up.
- Three fingers block.
Each of these is decided by counting which fingers are extended, which comes down to comparing fingertip positions with knuckle positions. Simple in theory; in practice half-open hands, motion blur and bad lighting produce ambiguous frames. The fix that made the game playable was to require a gesture to hold steady for a few frames before it fires. That small delay removed almost all accidental punches and is why the tracking guide tells you to hold each shape for a beat.
Always have a fallback
Early testers on phones showed us a problem: holding a phone and keeping a hand in front of its camera at the same time is awkward. So every input got a second path. Keyboard keys on laptops, on-screen touch buttons on phones, and a switch to turn hand tracking off entirely. Today a lot of phone players use the buttons and never touch the camera, and that is fine. The gestures are the game's signature, not a requirement.
Making it feel like a fighting game
A fighting game lives or dies on feel. The things that made the biggest difference were small: a few frames of freeze when a hit lands, a camera that nudges toward the action, a combo counter, and a health bar that shows the damage you just took as a fading ghost. The fighters are drawn with a small inverse-kinematics rig, so limbs bend at joints instead of sliding, and each character has real numbers behind them, which the roster guide lists in full.
The art went through a complete overhaul once. The first version was flat shapes; the current one uses a cel-shading pass with an outline and a gradient per body part. It runs in under half a millisecond per frame, which matters when a phone is already busy running the hand tracker.
Two players, one camera
Local two-player was easier than expected. The camera view is split down the middle: hands on the left half control player one, hands on the right control player two. Two friends stand side by side and fight. The only rule is to stay on your side.
Online play without a server
Online was the hardest part and the one we are proudest of. There is no game server. When you create a room, your browser registers a 4-letter code with a small signalling service, whose only job is to introduce two browsers to each other. After the introduction, the two browsers talk directly. The room creator's browser runs the match and decides every hit; the joiner sends inputs and receives the state. That keeps both screens in sync and makes cheating pointless.
Direct connections fail on strict networks, especially mobile ones, so there is a relay in the middle for those cases. That relay is the one thing about the online mode that costs money to run, and it is what the site's ads pay for. The online play guide explains what to do when a connection refuses to come up.
What we'd do differently
We would build the fallbacks first, not last. Every hour spent making the gestures more forgiving was worth more than an hour spent on effects. And we would test on cheap phones from day one, because the game that runs on a two-year-old budget Android is the game most of our players see.