Part of ProjectReachy Mini × Strands Agent on Jetson ThorA local-first voice & vision Reachy Mini Lite robot driven by an on-device Strands Agent on NVIDIA Jetson Thor

This is Part 3 of the series. In Part 2 I proved the hardware works — the head moves, the mic hears, the camera sees. Now the robot can start listening. This post covers the lowest layer of the voice stack: the "Hey Reachy" wake word, running fully offline with no LLM and no network.

While the assistant is idle, the only thing running is a small Vosk recognizer on the CPU. There is no Strands Agent, no Amazon Bedrock call, no Ollama inference, and no network traffic — so idle costs $0. Only when someone says "Hey Reachy" does the head raise and the per-wake agent spin up. This is what makes the whole build cheap to leave switched on: the expensive parts exist only between a wake and a reply.

The key idea — and the part that's trickier than it looks — is that "reachy" is not a word Vosk knows. It isn't in the English lexicon, so a constrained grammar built for the exact phrase would silently drop it and never fire. The wake detector instead runs full-vocabulary recognition and matches the homophones the model actually emits for "(hey) reachy", scanning both partial and final transcripts so it can trigger mid-sentence.

Goals

• Detect the wake phrase fully offline with Vosk — no cloud, no LLM, $0 while idle
• Reliably match a wake word that isn't in the recognizer's vocabulary, by matching its homophone renderings
• Trigger mid-utterance by scanning partial transcripts, not just finalized ones
• Capture the mic as raw ALSA PCM so the wake listener never fights the media daemon for the device
• On a match, raise the head as an "I'm listening" cue and hand off to the per-wake agent — unless a task is already running
• After a task, drop any audio captured during it so a stale utterance can't immediately re-trigger
• Ship a standalone demo (voice_wake.py) that proves the wake layer on its own, separate from the full assistant

The Overall System

The wake layer is a tight loop over raw mic audio. arecord produces 16 kHz mono PCM; Vosk turns chunks into partial and final transcripts; the idle loop scans them for the wake homophones; and on a match it raises the head and hands the conversation to handle_wake, which builds the per-wake Strands Agent (covered in later parts). Everything up to that hand-off is CPU-only and offline.

System Components:

1. arecord — captures the Reachy mic as raw S16_LE 16 kHz mono PCM; the daemon runs --no-media so the device is free for direct capture
2. Vosk (vosk-model-small-en-us-0.15, ~40 MB) — an offline recognizer that emits partial and final transcripts on the CPU
3. Idle loop (in reachy_assistant.py; the standalone demo is voice_wake.py) — reads PCM chunks, feeds Vosk, and matches WAKE_TOKENS
4. Reachy Mini Lite — raises its head on a wake as the "I'm listening" cue
5. Per-wake agent — only instantiated after a request is transcribed, the first time any LLM runs in the interaction

Interactive Sequence Diagram

Step through one wake, from idle listening to handing off to the agent and back to idle. Nothing before the hand-off touches an LLM or the network.

Offline Wake-Up: From Idle Listening to the Per-Wake Agent

CPU-only wake detection — no LLM, no network — until "Hey Reachy" is heard

0/9

Mic

Vosk

Loop

Robot

Agent

Milestone

Complete

9 steps across 5 components • CPU-only wake detection, no network

Idle is offline and $0 — only "Hey Reachy" spins up the agent

Architecture

The same wake logic exists in two places: a focused standalone demo and the embedded loop inside the full assistant.

Layer	File	Purpose
Standalone demo	voice_wake.py / voice_wake.sh	Robot sleeps head-down, wakes with a small gesture on "Hey Reachy", sleeps again — the wake layer provable on its own, zero LLM
Embedded loop	reachy_assistant.py	The idle listener that, on a wake, raises the head and hands off to handle_wake to build the per-wake agent

Both load the Vosk model once at startup, read raw PCM from arecord, and match a homophone token set. The standalone demo is the clearest way to see the layer in isolation; the assistant wires the same logic into the full per-wake lifecycle.

How it works

Raw PCM from arecord into Vosk

The mic stream is raw ALSA PCM, not the media daemon's audio path. The daemon is started --no-media precisely so the mic stays free for direct capture. The listener opens it via arecord:

subprocess.Popen(
    ["arecord", "-q", "-D", MIC_DEV, "-f", "S16_LE", "-r", "16000", "-c", "1", "-t", "raw"],
    stdout=subprocess.PIPE,
)

That is 16 kHz, signed 16-bit little-endian, mono, raw — exactly the format the Vosk recognizer expects. The model and recognizer are built once at startup, and MIC_DEV defaults to plughw:0:

rec = KaldiRecognizer(Model(VOSK_MODEL), 16000)   # load once

VOSK_MODEL points at vosk-model-small-en-us-0.15, which the entry script downloads once (~40 MB) on first run.

Matching a word Vosk can't spell

"reachy" is not in Vosk's English lexicon, so a constrained grammar for the exact phrase would drop the word and never fire. Instead the recognizer runs full-vocabulary, and the code matches the homophones the model actually emits for "(hey) reachy":

WAKE_TOKENS = ("reachy", "reach", "richie", "ritchie", "reachie", "reaches")

The loop reads the stream in 4000-byte chunks and feeds each to AcceptWaveform. Crucially it scans both partial and final transcripts, so the wake can trigger mid-utterance rather than waiting for a pause:

data = ar.stdout.read(4000)
if rec.AcceptWaveform(data):
    text = json.loads(rec.Result()).get("text", "")
else:
    text = json.loads(rec.PartialResult()).get("partial", "")
if text and any(tok in text for tok in WAKE_TOKENS):
    ...   # wake!

The standalone demo: sleep, wake, sleep

voice_wake.py makes the layer tangible. The robot rests head-down (goto_sleep()), and on a match it runs a small acknowledgement gesture — lift the head, a brief antenna wiggle, settle — then sleeps again and keeps listening:

def wake_gesture(mini):
    mini.enable_motors()
    mini.goto_target(INIT_HEAD_POSE, antennas=[0.6, 0.6], duration=0.5)
    mini.goto_target(create_head_pose(pitch=-8, degrees=True), antennas=[-0.3, -0.3], duration=0.4)
    mini.goto_target(INIT_HEAD_POSE, antennas=INIT_ANTENNAS_JOINT_POSITIONS, duration=0.5)

It connects with media_backend="no_media" (motors only) and prints what it transcribes ((heard: ...)) so you can see and tune what the model emits for your voice.

From a token match to listening

In the full assistant, a match is the entire trigger boundary. If a task is already running on the robot's worker the wake is ignored; otherwise the head raises as the "I'm listening" cue, and control hands off to listen_for_command, which transcribes one spoken request with a fresh recognizer — still offline, still $0:

if text and any(tok in text for tok in WAKE_TOKENS):
    if _busy.is_set():
        continue                       # busy — ignore this wake
    mini.enable_motors()
    mini.goto_target(INIT_HEAD_POSE, antennas=INIT_ANTENNAS_JOINT_POSITIONS, duration=1.0)
    request = listen_for_command(ar, vmodel)   # offline, $0

Only after a request is transcribed does handle_wake instantiate the per-wake Strands Agent. When the task finishes, the mic is restarted to drop any audio captured during the task, the recognizer is reset, and the loop returns to idle listening for the next "Hey Reachy".

Technical Challenges & Solutions

Challenge 1: The wake word isn't in the vocabulary

Problem: Vosk's small English model has no entry for "reachy". A constrained grammar built for "hey reachy" would never recognise a word it can't spell, so the wake would simply never fire.

Solution: Run full-vocabulary recognition and match the homophones the model actually produces — reachy, reach, richie, ritchie, reachie, reaches. The match is a substring test against that token set, which catches the real transcriptions instead of an idealised one. The standalone demo prints what it hears so the token set can be tuned to a given voice or accent.

Challenge 2: Waiting for a pause loses the wake

Problem: Vosk only finalises a transcript at a pause. If the listener only checked finalised results, "Hey Reachy, what's the weather" might not register the wake until the whole sentence ended — or not at all if the user runs straight on.

Solution: Scan partial transcripts as well as final ones. Each 4000-byte chunk is checked immediately, so the wake fires the moment the homophone appears mid-utterance, not after a pause.

Challenge 3: The wake listener vs. the media daemon

Problem: The mic is a single-opener device. If the daemon's media path owns it, a second reader can't capture audio for wake detection.

Solution: Start the daemon --no-media so the mic stays free, and capture it directly as raw PCM with arecord (or, when the media bus is running, subscribe to its mic broker — covered in a later part). The wake layer always has a clean audio source.

Challenge 4: Stale audio re-triggering after a task

Problem: While a task runs (the robot speaking, moving, looking), the mic keeps buffering. That backlog — including the robot's own speech or a trailing instruction — could immediately re-trigger a wake the instant the task ends.

Solution: After each task, the mic is restarted to discard the backlog and the recognizer is reset, so the loop returns to idle from live audio with a clean slate.

Getting Started

GitHub Repository: https://github.com/chiwaichan/nvidia-jetson-thor-strands-agent-reachy-mini-lite

Run the standalone wake demo

./voice_wake.sh    # downloads the Vosk model once, then listens for "Hey Reachy"

The robot rests head-down and lifts with a small gesture each time it hears the phrase. It prints what it transcribes so you can tune the token set to your voice.

Or the full assistant (same wake layer embedded)

./reachy_assistant.sh

This sets up the venv, installs the Reachy Mini Python SDK plus vosk, fetches the Vosk model on first run, starts a media-free daemon, and launches reachy_assistant.py. Say "Hey Reachy" to wake it; Ctrl-C to exit.

What's Next

In Part 4 - Local Vision with NVIDIA Cosmos Reason 2, I give the robot eyes: a Cosmos Reason 2 vision-language model running locally on the Jetson Thor GPU for scene description and visual Q&A — $0 per look — the tool the agent calls once a wake has handed it a question about the room.

Summary

This post covered the offline wake layer — the cheapest, lowest part of the voice stack:

• Offline and $0 idle — only a CPU-side Vosk recognizer runs until "Hey Reachy"; no LLM, Bedrock, Ollama, or network until a wake
• Matching an unknown word — "reachy" isn't in Vosk's lexicon, so the detector runs full-vocabulary and matches the homophones it actually emits (reachy, reach, richie, ritchie, reachie, reaches)
• Mid-utterance triggering — both partial and final transcripts are scanned, so the wake fires without waiting for a pause
• Clean audio source — the daemon runs --no-media and the listener captures raw S16_LE 16 kHz mono PCM directly via arecord
• Safe hand-off — a wake raises the head and calls handle_wake unless _busy is set; after a task the mic is restarted and the recognizer reset to drop backlog
• Provable in isolation — voice_wake.py is a standalone sleep/wake/sleep demo of the same logic, separate from the full assistant