A pose model hands you 33 body landmarks, thirty-ish times a second, each with a confidence score. Turning that stream into game controls that feel instant, never fire by accident, and work for a tall adult at three metres and a kid at one metre — that is where the real engineering lives. This article documents how CamPlay’s input engine actually does it, decision by decision, including the counterintuitive ones.
Decision zero: upper body only
The most surprising fact about CamPlay’s detection: it never looks at your legs. Of the 33 landmarks the model produces, the engine consumes exactly nine — nose, both shoulders, both elbows, both wrists, both hips. Nothing below the hips is required.
That is a product decision disguised as a technical one. Requiring ankles in frame means requiring the player to stand three or four metres from the laptop — which most bedrooms, dorms and offices do not allow. Requiring only head-to-waist means you can play about a metre from the screen. The cost is that “obvious” signals like knee height are off the table, and everything must be inferred from the torso and arms. The rest of this article is about paying that cost well.
The unit that makes everything work: shoulder width
Landmark coordinates are normalized to the camera frame, which means every raw distance changes when the player steps closer or farther away. A jump that displaces the nose by 12% of the frame at two metres displaces it 20% at one metre. Hard-coding pixel thresholds would make the controls feel different at every distance.
CamPlay’s fix is to measure everything in units of the player’s own smoothed shoulder width (SW). Shoulder width in the image scales with distance from the camera exactly the way every other body measurement does, so dividing by it cancels distance out. A jump threshold of “0.30 SW” means the same physical movement whether you are near or far, tall or small. This single normalization is what lets the engine ship without any calibration screen.
Jogging: detected from your arms and your head, not your feet
With no leg landmarks, how do you measure running in place? By its upper-body signature. When humans jog, three things happen in rhythm: the arms pump, the elbows rise with each swing, and the head bobs vertically. The engine multiplies all three together into a continuous speed value from 0 to 1:
- Cadence — a “pump” is counted when a wrist rises above its own slow-moving average while the same side’s elbow is also up. Around four pumps per second maps to full speed.
- Amplitude — bigger arm swings score higher than wrist wiggles, so half-hearted movement produces half-hearted speed.
- Head bob — the whole product is gated by vertical nose oscillation. This is the anti-cheat term: waving your arms with a still head scores roughly zero, because real jogging bounces your head and fake jogging doesn’t.
flowchart LR
W["Wrist rises above<br/>its slow average"] --> PUMP["Pump counted"]
E["Same-side<br/>elbow up"] --> PUMP
PUMP --> CAD["Cadence<br/>4 pumps/s ≈ full speed"]
CAD --> MUL(("×"))
AMP["Swing amplitude"] --> MUL
BOB["Head-bob gate<br/>still head ≈ zero"] --> MUL
MUL --> G["gamma 1.5"] --> SPD["speed 0–1"]The combined value is then gamma-compressed (raised to the power 1.5), which deliberately maps light jogging to low speeds — the difference between strolling pace and an actual sprint should be felt in the game, or there is no incentive to work. One practical wrinkle: wrists can leave the frame when arms hang fully down, so pump counting pauses and resumes per-arm without ever glitching the speed.
Jumps and ducks: two landmarks must agree
A naive jump detector — “nose moved up fast” — false-triggers constantly: on tiptoes, on stretches, on the natural head bounce of jogging. CamPlay requires two independent body parts to agree: a jump registers only when the nose rises more than 0.30 SW and the hips rise more than 0.25 SW together above their reference. Your head can move without your hips (looking up, stretching), but head and hips rising in sync means your whole body left the ground.
flowchart LR
N["Nose rises<br/>> 0.30 SW"] --> AND{"both,<br/>together"}
H["Hips rise<br/>> 0.25 SW"] --> AND
AND -- "agree" --> J["Jump event"]
AND -- "only one moves" --> X["Ignored<br/>(stretch, tiptoe, head bounce)"]Ducks are the mirror image: nose drops more than 0.55 SW and hips drop together. Jumps are one-shot events (fired once per hop); ducks are held states with separate enter and exit thresholds, so the game knows exactly how long you stayed low — which matters when you are sliding under a fire beam in Rune Run or holding a slow-motion dodge in Dino Dodge.
One more layer: the jump thresholds scale up with your current running speed. Jogging hard makes your head bounce more, so the bar to register a deliberate jump rises with it. Without that coupling, sprinting would fire phantom jumps every few strides.
The adaptive reference: how “rise” is defined without calibration
“Nose rises 0.30 SW” — above what, exactly? Above an adaptive reference: a slow-tracking average of where your nose and hips normally sit. It follows gradual change quickly — you drift closer to the camera, you slouch as you tire, the reference glides along — but it resists fast vertical motion, so the jump you are performing doesn’t drag its own baseline upward. And the moment a gesture begins, the reference freezes until the gesture ends, so holding a long duck can never “become the new normal”.
This is the piece that replaces the calibration step you might expect (“stand still with arms out”). There is no T-pose at startup, no stored per-user profile — the reference converges within a couple of seconds of you entering the frame and re-converges after any posture change.
Leans and lanes: intent versus position
Sideways movement carries two different meanings, so the engine tracks them as two different signals. A lean is an intent gesture: it registers when your shoulder line rolls past 18 degrees and your head shifts sideways more than 0.22 SW past the shoulder midpoint — both required, so glancing sideways doesn’t steer you. It releases at 10 degrees; the 8-degree gap (hysteresis) prevents flicker when you hover near the threshold. In Ski Slalom the same roll angle is used as a fully analog value — a gentle lean carves a long arc, a hard lean cuts tight.
A lane, by contrast, is a position: which third of the frame your body occupies. The boundaries are not straight vertical lines — they are slanted like parking-camera guides, making the centre lane narrower at the top of the frame and wider at the bottom, which matches how a body projects into the image as it moves sideways. Your lane is whichever region overlaps most of your body’s bounding box at torso height, and switching requires beating the current lane’s overlap by an 8% margin — again hysteresis, so standing on a boundary never rattles you between lanes.
Stillness as a control: the Red Light problem
One game inverts the whole problem. Red Light, Green Light needs to know you are NOT moving — and “no jump events fired” is far too weak a definition when the doll is staring at you. For this, the engine exposes a raw motion signal: the velocity of your fastest-moving tracked points, in shoulder-widths per second. During a red light the game layers several tests on it: a fast-rule for outright running, a slow-burn rule for sneaky low-speed creeping sustained over a second, the raw motion signal held above threshold, and any discrete gesture at all. A short grace period after the light turns covers honest human reaction time, then the judge goes lethal.
Knowing when NOT to trust yourself
The engine’s most important signal is the humblest: ready. It is true only while all nine required landmarks are confidently visible — and it must hold for 10 consecutive frames to switch on, but drops after 15 bad frames. While not ready, input is paused, held states release, and speed decays to zero. The games surface this immediately (“step back into frame”) rather than letting a half-visible body produce garbage controls. In multiplayer, staying untrackable for more than 1.2 seconds disqualifies the run — because a player the camera cannot see is a player the game cannot referee. The same philosophy runs through the calorie tracker: if the camera can’t see you, you earn nothing.
The shape of the whole thing
Every technique above is a variation on three principles. Normalize by the body itself (shoulder width), so nothing depends on where the player stands. Demand agreement between independent signals (nose AND hips, roll AND head shift, cadence AND bob), so no single noisy landmark can trigger anything. And put hysteresis on every boundary, so thresholds feel decisive instead of jittery. None of it is exotic machine learning — the model does the perception, and everything on top is careful, testable signal engineering. That split is what makes the system debuggable: when a control misbehaves, the fix is a threshold with a name, not a retraining run.