Correction inside a frame.
Feedback later than about ninety milliseconds reads as judgement, not guidance. The budget was one render frame.
Case 02 — Spatial computing
Sign of Love
A Vision Pro app that watches your hands and corrects them as you learn to sign. The entire product rests on one question: can the device tell a mistake from a moment it simply could not see?
Fig. 00 — the kuji-in study, where it started
The ask
Teach American Sign Language with the feedback a human tutor gives, without the tutor.
Video lessons can show you a sign. They cannot tell you that your thumb is two centimetres too low, that your movement started before your handshape had settled, or that the sign you just produced means something close to, but not, what you intended.
Human tutors can do all of that, and they are scarce, expensive and rarely available at eleven at night when a learner has fifteen minutes. Spatial computing is the first consumer platform where a machine can see a three-dimensional gesture well enough to comment on it. That made it worth attempting, and hard in exactly one place.
What had to be trueThree constraints, none negotiable
Trust the edge of view.
Tracking confidence collapses at the periphery, exactly where signing happens. We mapped where, and designed the lessons around it.
Wrong, gently.
When confidence drops the app says less and never guesses. Silence beats a false correction.
The hard part
Hand-tracking confidence collapses at the edges of the field of view, and a low-confidence frame looks exactly like a badly formed sign.
Signing is not performed politely in front of your face. ASL uses the space beside the body, above the head and close to the chest. A meaningful fraction of correct signing happens precisely where the headset's cameras see worst.
A naïve implementation punishes the learner for the hardware's limitations. Tell someone their signing is wrong when it was in fact perfect and merely unobserved, and you have not built a tutor. You have built something that erodes confidence in a domain where confidence is the entire point.
Fig. 02 — Tracking confidence against gaze-relative angleIllustrative model
Reported confidence
Naïve threshold — a single cutoff
Where correct ASL actually happens
A single confidence threshold puts the cutoff in the middle of the region where legitimate signing occurs. Everything to the right of the dashed line gets marked wrong. The fix was never a better threshold. It was refusing to make a judgement from a single frame.
The approach
Separate "wrong" from "unseen". Never guess.
We stopped treating each frame as evidence and started treating a sign as an interval. The recogniser accumulates observations across the duration of a gesture, weights them by the confidence the system reports for that frame, and produces a judgement only when the accumulated evidence crosses a bar, with an explicit third outcome alongside correct and incorrect: not enough signal.
That third state changed the product. When the system cannot see well enough, it does not stay silent and it does not blame the learner. It says so, and it moves the lesson content, rendered in space and therefore movable, to where the hands can actually be seen. The correction and the camera geometry became the same problem.
WEEKS 1–3
Prototype the risk
Before any lesson design, a bare app that logged tracking confidence against hand position while a fluent signer signed normally. The data set the whole architecture.
Running buildWEEKS 4–9
Interval recogniser
Confidence-weighted accumulation over gesture duration, with the three-state output. Tuned against recordings from signers of different heights, hand sizes and speeds.
Core engineWEEKS 10–15
Lessons that move
Content anchored where the cameras see best, repositioning quietly when tracking degrades. The pedagogy and the perception budget designed together, not in sequence.
ProductWEEKS 16–19
Harden and ship
Thermals, session length, accessibility, review guidelines for a brand-new store category, and the failure modes that only appear on a headset worn for forty minutes.
App StoreThe outcomeNumbers we will defend
19
Weeks, sketch to App Store
<90ms
Sign to feedback
90fps
Held, throughout
3
Outcome states, not two
What we would do differently.
01
Record the reference data sooner. We built the capture harness in week two and wished it had been week zero. Every architectural decision downstream depended on it, and three days earlier would have saved a fortnight of re-tuning.
02
Bring a Deaf consultant in at week one, not week six. We got the engineering right and initially got some of the pedagogy wrong. Domain expertise is not a review step you schedule near the end.
03
Budget thermals as a feature, not a bug. Session length on a headset is governed by heat. We treated that as a late-stage constraint when it should have shaped the lesson structure from the beginning.
What it taught us
Knowing when the machine cannot know is the whole company now.
The third state is the rule behind everything we have built since. Reach's known-anomalies list says what it cannot guarantee on which OS. Chart makes no alerts, because sensor data reaches Apple Health hours late. A Reach Report names a device and a build, or it says nothing.
The hand-tracking research started as Kuji-Vision on the bench. The recogniser shipped here. The habit of saying "not enough signal" shipped everywhere.
Reach, the alert layerThe third state, "I could not see that", is the whole product. Everything else is a recogniser.
Next case