Part of ProjectReachy Mini × Strands Agent on Jetson ThorA local-first voice & vision Reachy Mini Lite robot driven by an on-device Strands Agent on NVIDIA Jetson Thor

This is Part 5 of the series — and the centerpiece. Part 3 gave the robot ears and Part 4 gave it eyes. This post is the loop that ties them — and everything still to come — together: "Hey Reachy" → transcribe → a fresh per-wake Strands Agent that picks a tool → speak one short sentence → tear down.

The design goal is a robot you can leave switched on without it costing anything to sit idle. So the loop is cost-minimal by construction: while waiting, only the offline Vosk wake listener runs — no agent, no model loaded into the loop, no backend calls, $0. The Strands Agent exists only between a wake and a reply. On wake it's built fresh around the one request, it reasons with its model, calls exactly the tool it needs, speaks a single sentence, and is then deleted and garbage-collected. Nothing LLM-related survives to the next wake.

Two trigger sources feed one worker — the voice wake word and an optional AWS IoT Core MQTT subscription — so the robot has a single owner and is never driven by two sources at once.

Goals

• Tie the wake layer, vision, motion, emotions, and data-lake tools into one wake → think → act → speak loop
• Keep idle at $0 — only the offline Vosk listener runs until a wake
• Build a brand-new agent per wake and destroy it afterwards, so no LLM process persists between interactions
• Give the agent its full twelve-tool surface and let it pick the right one for the request
• Bound every wake with a hard ModelCallBudget so cost and latency can't run away
• Reply with one short spoken sentence via local TTS (Piper → espeak-ng → print), never raw JSON
• Accept voice and MQTT triggers through one worker that owns the robot
• Print a per-wake cost/latency summary so every interaction is accountable

The Overall System

The loop is a pipeline with a single robot owner at its center. A trigger (voice or MQTT) enqueues a request; one worker thread drains the queue; handle_wake builds a fresh agent; the agent reasons and calls a tool; the reply is spoken; the agent is torn down; and the loop returns to idle.

System Components:

1. Triggers — the offline Vosk wake word and an optional AWS IoT Core MQTT subscription, both enqueuing to one queue
2. Worker — a single thread that drains the queue so the motors and agent are never driven by two sources at once
3. handle_wake — builds the fresh per-wake Agent, runs it under the budget, speaks the reply, tears it down
4. Model — local Nemotron via Ollama ($0) by default, or Amazon Nova 2 Lite on Bedrock
5. Twelve tools — vision, six motion primitives, emotions (×2), and three data-lake tools
6. Speaker — local TTS via Piper, falling back to espeak-ng, then print

Interactive Sequence Diagram

Step through one full wake — trigger, route, build, reason, tool, speak, tear down — and back to idle.

The Per-Wake Loop: Wake to Reply to Teardown

A fresh agent is built, picks one tool, speaks, and is destroyed — then back to $0 idle

0/10

Trigger

Worker

Agent

Model

Tool

Speak

Milestone

Complete

10 steps across 6 components • agent exists only between wake and reply

Wake -> one tool -> one sentence -> teardown, then $0 idle

Architecture

handle_wake(request) is the per-wake core. Every wake it constructs an Agent with the full toolset and a cost guard, runs it, and destroys it:

budget = ModelCallBudget(MAX_MODEL_CALLS)
agent = Agent(
    model=_build_model(),
    system_prompt=sys_prompt,
    tools=[look_and_describe, play_emotion, list_emotion_moves,
           nod, shake_head, look_around, wiggle_antennas, spin_body, move_head,
           list_iot_tables, get_table_schema, query_iot_data],
    hooks=[budget],
)
result = agent(request)

The twelve tools span every capability in the series:

Group	Tools	Covered in
Vision	look_and_describe	Part 4
Motion	nod, shake_head, look_around, wiggle_antennas, spin_body, move_head	Part 1
Emotion	play_emotion, list_emotion_moves	Part 8
Data lake (S3 Tables)	list_iot_tables, get_table_schema, query_iot_data	Part 6

The system prompt steers the agent to discover-before-guess and to always reply with one short spoken sentence — never raw JSON, table dumps, or column lists.

How it works

Idle costs nothing

While waiting, only an offline Vosk wake-word recognizer runs against the mic stream. No Strands Agent exists, no model is loaded into the loop, and nothing is sent to a backend — the idle path is $0. A second trigger, an AWS IoT Core MQTT subscription, enqueues to the same worker, so a wake can come from voice or a published message. Both feed one worker that owns the robot.

On wake: head up, then transcribe offline

When a wake token matches and the worker is not already busy, main enables the motors and raises the head to neutral as the "I'm listening" cue, then transcribes the request — still fully offline, still $0, before any backend is touched:

mini.enable_motors()
mini.goto_target(INIT_HEAD_POSE, antennas=INIT_ANTENNAS_JOINT_POSITIONS, duration=1.0)
request = listen_for_command(ar, vmodel)   # offline, $0

listen_for_command reuses the running arecord stream but builds a fresh KaldiRecognizer so the wake word itself isn't carried into the request. It returns when Vosk reports end-of-utterance with non-empty text, or after LISTEN_SECONDS (default 8) as a backstop.

A brand-new Strands Agent per wake

The model comes from _build_model(): by default LLM_BACKEND=ollama runs a local Nemotron model through Ollama on the Thor GPU ($0); LLM_BACKEND=bedrock runs Amazon Nova 2 Lite on Bedrock instead. ModelCallBudget is a HookProvider registered on BeforeModelCallEvent; it counts every model call and raises once the count passes MAX_MODEL_CALLS (default 12), hard-capping a single wake. The cap is set high enough for a data-lake question, whose discover → schema → query → answer cycle needs several round-trips.

The request reaching the agent is routed first: a spoken request beginning with the emotion prefix becomes a play_emotion instruction; otherwise it passes through unchanged so the agent stays generic.

One short spoken sentence, then teardown

The agent's final string is cleaned, spoken, and accounted for:

result = agent(request)
_print_summary(result, budget, t_wall)
reply = _clean_reply(str(result))
speak(reply)

_print_summary prints the backend and model, model calls : <n> / MAX_MODEL_CALLS, token in/out/total counts, an estimated LLM cost ($0 for local Nemotron), latency, cycle count, and wall time — a per-task accounting of exactly what the one wake spent. speak does local TTS through the Reachy speaker: a Piper voice if one loaded at startup (synth a WAV, play via aplay), falling back to espeak-ng, then to printing the reply.

Teardown runs in a finally so it always happens:

del agent          # destroy the agent instance
gc.collect()

Nothing LLM-related survives between wakes. The head deliberately stays up (_head_up) rather than dropping to sleep, and the loop returns to idle Vosk-only listening at $0.

Technical Challenges & Solutions

Challenge 1: Keeping idle free

Problem: A always-on robot assistant that holds a model resident or polls a backend would cost money and power to do nothing.

Solution: The agent is built only inside handle_wake and destroyed in a finally. Between wakes, only the CPU-side Vosk listener runs — no agent, no loaded model in the loop, no backend traffic. Idle is genuinely $0, verified by the per-wake summary that reports $0 for local-backend runs.

Challenge 2: Two trigger sources, one robot

Problem: Voice and MQTT can both arrive at any time. If both drove the robot directly, two tasks could move the motors or call the agent at once.

Solution: Both triggers enqueue onto one queue drained by a single worker thread. The robot has exactly one owner; a wake heard mid-task is ignored, and the MQTT callback is fire-and-forget so its keep-alive never stalls.

Challenge 3: A runaway agent loop

Problem: A tool-calling agent can loop — reason, call, reason, call — and on a cloud backend that's unbounded cost; on the robot it's unbounded motion.

Solution: ModelCallBudget caps model calls per wake (default 12) and aborts cleanly when exceeded. The cap is generous enough for the longest legitimate flow (a data-lake question's discover → schema → query → answer) but still bounds the worst case.

Challenge 4: Speaking, not dumping data

Problem: Tools return JSON, table rows, and column lists. Read aloud, that's unusable.

Solution: The system prompt requires the agent to answer with one short, natural spoken sentence and never read raw JSON or tables, and _clean_reply strips any <think> reasoning a reasoning model emits before it reaches speak.

Getting Started

GitHub Repository: https://github.com/chiwaichan/nvidia-jetson-thor-strands-agent-reachy-mini-lite

Run it

./reachy_assistant.sh    # say "Hey Reachy"; Ctrl-C to stop

reachy_assistant.sh creates the venv; installs reachy-mini, vosk, strands-agents, and piper-tts; downloads the Vosk model and Piper voice once; starts the daemon with --no-media --no-wake-up-on-start (so the controller owns the head and keeps it up for the session); and exports MAX_MODEL_CALLS, LLM_BACKEND, and the model ids before launching reachy_assistant.py. Say "Hey Reachy" and give it a request — a look, a gesture, an emotion, or a data-lake question.

What's Next

In Part 6 - IoT Data-Lake Q&A on S3 Tables, I wire the agent to an AWS data lake (S3 Tables / Apache Iceberg) through Lambda and Athena, so "Hey Reachy, has the kitchen water sensor tripped today?" becomes a discover → schema → query → answer flow spoken back in one sentence.

Summary

This post covered the centerpiece loop that turns the parts into an assistant:

• Wake → think → act → speak — offline wake, offline transcription, a fresh per-wake agent that picks one of twelve tools, then one spoken sentence
• $0 idle — only the Vosk listener runs between wakes; the agent is built in handle_wake and destroyed in a finally (del agent; gc.collect())
• Pluggable brain — local Nemotron via Ollama by default, Amazon Nova 2 Lite on Bedrock as an opt-in swap
• One robot owner — voice and MQTT both enqueue to a single worker, so two sources never drive the motors at once
• Bounded cost — ModelCallBudget caps model calls per wake (default 12), generous enough for a data-lake question but never unbounded
• Accountable — a per-wake summary prints backend, model calls, tokens, estimated cost, latency, and wall time; local TTS speaks the reply via Piper → espeak-ng → print