Sign of Love
An Apple Vision Pro app that corrects your hands as you learn American Sign Language. It rests on one question: can the device tell a mistake from a moment it could not see?
A tutor’s feedback, without the tutor.
Video lessons show you a sign. They cannot tell you your thumb is two centimetres low.
Tutors can, but rarely at eleven at night. Apple Vision Pro is the first consumer device that sees a gesture well enough to comment on it. It was hard in exactly one place.
| Inside a frame | Corrections within a frame. Feedback later than about ninety milliseconds feels like judgement, so each correction had to land within a single frame. |
|---|---|
| Edge of view | Trust the edge of view. The headset sees hands worst exactly where signing happens. We mapped it and designed the lessons around it. |
| Wrong, gently | Silence beats a false correction. When confidence drops the app says less and never guesses. |
At the edges of its view, a hand the headset cannot see looks just like a badly formed sign.
And signing happens beside the body, above the head and close to the chest: exactly where the cameras see worst.
A naïve build punishes the learner for the hardware: a perfect sign, unseen, gets marked wrong. That is not a tutor.
One threshold cuts through the middle of real signing. The fix was never a better threshold. It was refusing to judge one frame.
Separate “wrong” from “unseen”. Never guess.
We stopped judging single frames and judged the whole sign, trusting each moment as much as the headset was sure of it. Beside correct and incorrect there is a third answer: I could not see that.
That answer changed the product. When the app cannot see, it says so and moves the lesson to where the hands can be seen. Teaching and camera angles became one problem.
Prototype the risk
Before any lessons, we recorded how sure the headset was of each hand while a fluent signer signed. That data shaped everything.
Running buildJudge the whole sign
A recogniser that weighs the whole gesture and answers in three ways, tuned on signers of different heights, hand sizes and speeds.
Core engineLessons that move
Lessons placed where the cameras see best, moving when tracking slips. Teaching and tracking, designed together.
ProductHarden and ship
Heat, session length, accessibility, App Store review for a brand-new category, and the problems that only appear after forty minutes in a headset.
App StoreWhat we would do differently.
Record the reference data sooner. The recording tool came in week two and should have been week zero. Every decision downstream depended on it.
Bring a Deaf consultant in at week one, not six. We got the engineering right and some of the pedagogy wrong. Domain expertise is not a review step.
Budget thermals as a feature. Session length on a headset is governed by heat. It should have shaped the lesson structure from the start.
Now everything we build says what it cannot know.
Say what the system cannot know, on which device, and stop there. Reach is built on it: what it cannot guarantee, on which OS, is written down.
The hand-tracking research started as Kuji-Vision, one of our experiments in Labs. The recogniser shipped here. The habit of saying “not enough signal” shipped everywhere.
See ReachThe third state, “I could not see that”, is the whole product. Everything else is a recogniser.
| Made by | The team at Bolted. |
|---|---|
| Platform | visionOS, on Apple Vision Pro. |
| Links | signoflove.net |
Otta
Mac app · Our own
