Reachy Mini x Strands Agent on Jetson Thor - Part 1: Driving the Reachy Mini Lite with Strands Agent
This is Part 1 of a series that builds a Reachy Mini Lite desk robot driven by a Strands agent running entirely on-device on an NVIDIA Jetson Thor. Say "Hey Reachy" (or publish an MQTT message) and a fresh agent wakes up, decides which tool it needs — move its head and body, see the room with a local vision model, express an emotion, or answer questions about IoT sensor data in an AWS data lake — speaks one short sentence, and tears itself down. Idle is pure-local and $0: no cloud, no LLM tokens, just an offline wake-word listener.
This post covers the piece that came first and that everything else grew from: the motion foundation. A Strands Agent is given a handful of motion tools that wrap the Reachy Mini Python SDK, and it decides which ones to chain together to act out a plain-English instruction on the physical robot over USB. Everything else in the series — vision, voice, the data lake, emotions, telemetry — is layered on top of this same fresh-agent-per-wake pattern.
The key idea is that the agent is never hard-coded to a fixed routine. Given "nod twice, wiggle your antennas, then spin around", the LLM reads the request, picks the matching motion primitives, and calls them one at a time, in order, before it replies — so it can compose novel gestures the code never explicitly programmed. The whole run is bounded by a hard model-call budget so a runaway loop can never rack up cost or keep driving the motors indefinitely, and the robot is always left in a safe neutral pose afterwards.
The series is a single repository (nvidia-jetson-thor-strands-agent-reachy-mini-lite) built up capability by capability. The next post — Part 2 - Hardware Bring-Up — covers the one-time host setup (USB udev permissions, the GStreamer media plugin, the daemon, and the SDK connection) that lets the agent reach the robot at all. Later parts add offline voice wake-up, local vision with Cosmos Reason 2, the full per-wake assistant loop, IoT data-lake Q&A, an MQTT trigger, emotion moves, and more.
Goals
- Give a Strands Agent a set of motion primitives —
nod,shake_head,look_around,wiggle_antennas,spin_body,move_head— that wrap the Reachy Mini Python SDK and drive the robot's head, body, and antennas - Let the LLM compose several tools to act out a multi-step instruction, performing every step in order, one tool call per step, before it replies
- Run each instruction through a brand-new agent that is built, used, then destroyed — so no long-lived LLM process burns resources between interactions
- Cap model calls per run with a hard
ModelCallBudgetStrands hook so a runaway tool-call loop can never rack up unbounded cost or keep the motors moving - Make every tool fail soft — a hardware error becomes a status string the agent reads back, never a crash
- Always leave the robot in a safe, known neutral pose, whether the task finished, raised, or tripped the budget
- Keep the reasoning model pluggable: local Nemotron via Ollama ($0, offline) by default, or Amazon Nova 2 Lite on Amazon Bedrock as an opt-in swap — identical tools and prompt either way
The Overall System
The motion foundation is a short, closed loop. A plain-English instruction enters the Strands Agent; the agent reasons with its model (local Nemotron or Bedrock Nova 2 Lite) about which motion tools to call; each tool wraps a Reachy Mini Python SDK call and drives the robot through the local reachy-mini-daemon over USB; and each tool returns a short status string the agent reads back before deciding its next move.
System Components:
- Instruction — a plain-English request, either spoken ("Hey Reachy, nod twice then spin") or delivered over MQTT, handed to the agent
- Strands Agent (
reachy_assistant.py) — a freshAgentbuilt per wake with the motion tools, a system prompt, and theModelCallBudgethook - Reasoning model — local Nemotron via Ollama by default ($0, offline), or Amazon Nova 2 Lite on Bedrock as an opt-in backend; both decide which tools to call
- Motion tools — ordinary Python functions decorated with
@toolthat wrap the Reachy Mini Python SDK, clamp their arguments, and return a status string reachy-mini-daemon— owns the USB link to the robot and exposes its motors; the SDK talks to it over localhost- Reachy Mini Lite — the physical robot: a 6-DOF head, a body that rotates about a vertical axis, and two antennas
Interactive Sequence Diagram
Step through one wake end to end — the worked example "nod twice, then spin around". The agent reasons, selects a tool, drives the robot through the daemon, reads back the status string, reasons again for the next step, and finally speaks and tears itself down at neutral. The markers are the real sequence (model-call number, tool name), not measured latencies.
Per-Wake Motion Loop: Instruction to Servo Movement
A fresh agent composes motion tools step by step, then tears itself down at neutral
Architecture
The agent is a Python application built on the Strands Agents framework. On each wake the request is handed to handle_wake(), which builds a fresh Agent around it, runs it under the model-call budget, and then destroys it. A single ReachyMini connection is opened once at startup and shared by every tool.
The per-wake agent
budget = ModelCallBudget(MAX_MODEL_CALLS)
agent = Agent(
model=_build_model(),
system_prompt=sys_prompt,
tools=[..., nod, shake_head, look_around, wiggle_antennas, spin_body, move_head, ...],
hooks=[budget],
)
result = agent(request)
The system prompt tells the agent it has a physical body — a 6-DOF head, a body that rotates around a vertical axis, and two antennas — and that for literal or directional motion it should call the motion primitives, performing every step of a multi-part request in order, one tool call each, before it replies.
In the full assistant the agent is handed a broader toolset (vision, emotions, and data-lake tools, covered in later parts). This post focuses on the six motion primitives that form the foundation — the part the agent uses to move its body.
Motion tools
Each tool is an ordinary Python function decorated with @tool. It wraps a Reachy Mini Python SDK call (goto_target), clamps its arguments to a safe range, returns to neutral where it makes sense, and returns a short status string that the agent reads back as the tool result.
| Tool | Behavior | Clamp |
|---|---|---|
nod(times=2) | Pitch the head up/down to say "yes", then return to neutral | 1–5 |
shake_head(times=2) | Yaw the head left/right to say "no", then return to neutral | 1–5 |
look_around() | Sweep yaw +60 → −60 → center to scan the room | — |
wiggle_antennas(times=3) | Both antennas up/down expressively, then return to neutral | 1–6 |
spin_body(degrees=90, duration=1.5) | Rotate the body about its vertical axis | ±160° |
move_head(pitch, roll, yaw, duration=1.0) | Absolute head orientation for a deliberate look/tilt | pitch/roll ±40°, yaw ±180° |
nod, shake_head, and move_head build target poses with create_head_pose. The neutral-pose constants INIT_HEAD_POSE and INIT_ANTENNAS_JOINT_POSITIONS, which the gesture tools return to, also come from the Reachy Mini Python SDK. The six tools began life as a standalone demo (agent_demo.py) and are now first-class tools in reachy_assistant.py — the pattern everything else grew from.
How it works
A tool wraps the SDK and fails soft
nod is representative of all six. It clamps times to 1–5, builds head poses with create_head_pose, drives the motors with goto_target, returns the head to INIT_HEAD_POSE, and reports back as a plain string. The whole body is wrapped so a connection error or hardware fault becomes a status string the agent can read rather than an exception that crashes the run:
@tool
def nod(times: int = 2) -> str:
"""Nod the head up and down to say 'yes' or to acknowledge."""
if mini is None:
return "Robot not connected."
try:
n = max(1, min(int(times), 5))
for _ in range(n):
mini.goto_target(create_head_pose(pitch=15, degrees=True), duration=0.35)
mini.goto_target(create_head_pose(pitch=-10, degrees=True), duration=0.35)
mini.goto_target(INIT_HEAD_POSE, duration=0.35) # back to neutral
publish_state("motion", motion="nod", times=n) # telemetry (later part)
return f"Nodded {n} time(s)."
except Exception as e:
return f"Could not nod: {e}"
spin_body and move_head use a _clamp helper instead of returning to neutral — spin_body clamps to ±160°, and move_head clamps pitch/roll to ±40° and yaw to ±180° — so the LLM can never command the servos past a safe range no matter what it passes. Each tool also publishes a motion state snapshot to IoT Core; that telemetry is a no-op here and is covered in a later part.
The system prompt steers the composition
The agent isn't given canned routines. The system prompt describes the body and tells it, for literal or directional motion, to call the primitives — and crucially, when a request lists several motions, to perform all of them in order, one tool call each, before replying:
To MOVE or gesture literally, call the motion tools: nod (yes), shake_head (no), look_around (scan the room), wiggle_antennas, spin_body (turn the body), or move_head (a deliberate look/tilt). If the request lists SEVERAL motions, perform ALL of them in order, one tool call each, before you reply.
So "nod twice, then look left, then spin" becomes three deliberate tool calls — nod, then move_head, then spin_body — chained by the agent loop, each one's status string feeding the next decision.
ModelCallBudget — a hard ceiling per run
ModelCallBudget is a Strands HookProvider that puts a hard ceiling on model calls per agent invocation. It registers a BeforeModelCallEvent callback that increments a counter on each agent-loop turn and raises a RuntimeError once the cap is exceeded, so a runaway loop can never rack up unbounded cost (or keep driving the motors indefinitely):
def _before(self, _e: BeforeModelCallEvent) -> None:
self.count += 1
print(f" ↳ Strands agent → {LLM_BACKEND} model call #{self.count}/{self.max_calls} ({ACTIVE_MODEL})")
if self.count > self.max_calls:
raise RuntimeError(f"Model-call budget exceeded ({self.max_calls}).")
The ceiling is MAX_MODEL_CALLS (default 12 — enough headroom for a multi-step gesture or a discover→schema→query→answer data-lake question, covered later).
A pluggable brain — Nemotron or Bedrock
The reasoning model is chosen by one environment variable. By default the agent runs on local Nemotron via Ollama ($0, offline); set LLM_BACKEND=bedrock to swap in Amazon Nova 2 Lite on Bedrock. The motion tools and system prompt are identical either way:
def _build_model():
if LLM_BACKEND == "bedrock":
return BedrockModel(model_id=MODEL_ID, region_name=AWS_REGION)
from strands.models.ollama import OllamaModel
return OllamaModel(host=OLLAMA_HOST, model_id=NEMOTRON_MODEL)
Going fully local — the Jetson-resident Nemotron and the gotchas of swapping a reasoning model in for Bedrock — is a topic of its own later in the series.
Fresh agent per wake, then torn down
Each wake builds a brand-new agent and destroys it afterwards, so there is no long-lived LLM process holding state (or cost) between interactions. handle_wake() runs the agent inside a try/finally, and after the task — whether it finished, raised, or tripped the model-call budget — the agent is deleted and garbage-collected, then _head_up() drives the robot back to the upright neutral pose:
finally:
del agent # destroy the agent instance
gc.collect()
_head_up() # return to upright neutral (head stays up, not asleep)
The head is kept up between wakes (rather than dropped to sleep), so the robot always rests in a known, safe pose ready for the next instruction.
Technical Challenges & Solutions
Challenge 1: Runaway model-call cost
Problem: An agent that decides its own tool calls can loop — calling a tool, reasoning, calling another, indefinitely — which on a cloud backend racks up unbounded cost and on the robot keeps the motors moving with no end.
Solution: The ModelCallBudget Strands hook counts every BeforeModelCallEvent and raises a RuntimeError once MAX_MODEL_CALLS (default 12) is exceeded. The run aborts cleanly, the finally block still recenters the robot, and a single env var tunes the ceiling per deployment.
Challenge 2: Leaving the robot in a safe pose, always
Problem: A gesture can leave the head tilted or the antennas raised, and a task can end abnormally — an exception, a tripped budget, or a Ctrl-C — mid-movement, stranding the robot in an awkward or unsafe pose.
Solution: Two layers. Each gesture tool ends by sending the relevant joints back to INIT_HEAD_POSE / INIT_ANTENNAS_JOINT_POSITIONS. Beyond that per-tool cleanup, handle_wake() runs the agent inside try/finally, and after every task _head_up() drives the robot back to the upright neutral pose — so it always rests in a known state ready for the next wake.
Challenge 3: A hardware fault shouldn't crash the agent
Problem: The robot is real hardware over USB — a servo can be busy, the connection can drop, an SDK call can raise. An unhandled exception inside a tool would tear down the whole agent run.
Solution: Every tool first checks that mini is connected and wraps its SDK call in try/except, turning any failure into a short status string ("Could not nod: ..."). The agent reads that back as the tool result and can react or move on, rather than crashing.
Challenge 4: Composing novel multi-step gestures
Problem: Users ask for things the code never explicitly programmed — "shake your head no, then wiggle your antennas, then look left" — and a naive agent might collapse that into a single action or reply before finishing.
Solution: The system prompt instructs the agent that when a request lists several motions, it must perform all of them, in order, one tool call each, before replying. The six primitives plus that instruction let the agent assemble arbitrary multi-step gestures from a small, safe vocabulary — the foundation the rest of the build composes on.
Getting Started
GitHub Repository: https://github.com/chiwaichan/nvidia-jetson-thor-strands-agent-reachy-mini-lite
Prerequisites
- Reachy Mini Lite assembled and connected over USB, with the motor power supply on
- NVIDIA Jetson Thor (or another box) — for the default local Nemotron backend you need Ollama serving a tool-calling model; for the Bedrock backend you need AWS credentials with Nova 2 Lite access
- One-time host setup (USB udev permissions and the GStreamer media plugin) — covered in Part 2
Running it
reachy_assistant.sh is the entry point. It bootstraps the environment (installs uv, creates the venv, installs reachy-mini and strands-agents), starts reachy-mini-daemon --no-media if it is not already up, then launches reachy_assistant.py:
./reachy_assistant.sh
Then say "Hey Reachy" and give it an instruction — for example "nod twice, wiggle your antennas, then spin around" — and the agent chains the matching motion tools to act it out. Behaviour is configured through env vars: LLM_BACKEND (ollama or bedrock), BEDROCK_MODEL_ID, AWS_REGION, and MAX_MODEL_CALLS.
What's Next
In Part 2 - Hardware Bring-Up, I cover the one-time host setup that lets the agent reach the robot at all: the USB udev permissions, the GStreamer webrtcsink plugin, the daemon, and the Reachy Mini Python SDK connection — plus the silent-mic ribbon-cable gotcha that trips up a fresh build.
From there the series layers capability onto this same foundation: offline voice wake-up with Vosk, local vision with NVIDIA Cosmos Reason 2, the full per-wake assistant loop, IoT data-lake Q&A through Athena and Iceberg, an AWS IoT Core MQTT trigger, the ~80-move emotion library, and going fully local on Nemotron via Ollama.
Summary
This post covered the motion foundation — the part of the build everything else grows from:
- Strands Agents framework with a fresh agent per wake — built, used, then
del'd and garbage-collected so no long-lived LLM process holds state or cost between interactions - Six motion primitives (
nod,shake_head,look_around,wiggle_antennas,spin_body,move_head) that wrap the Reachy Mini Python SDK, clamp their arguments to a safe range, and return a status string the agent reads back - Composable gestures — the system prompt steers the agent to perform every step of a multi-part request in order, one tool call each, so it assembles novel multi-step motions from a small, safe vocabulary
ModelCallBudgethook caps model calls per run (default 12) so a runaway loop can never rack up unbounded cost or keep the motors moving- Fail-soft tools — every SDK call is wrapped so a hardware fault becomes a status string, never a crash — and the robot is always left in a safe neutral pose via per-tool cleanup plus a
finally+_head_up() - Pluggable brain — local Nemotron via Ollama ($0, offline) by default, or Amazon Nova 2 Lite on Bedrock as an opt-in swap, with identical tools and prompt either way