Part of ProjectReachy Mini × Strands Agent on Jetson ThorA local-first voice & vision Reachy Mini Lite robot driven by an on-device Strands Agent on NVIDIA Jetson Thor

This is Part 9 of the series. So far the robot only acts when triggered — a wake word or an MQTT message. Now I want it to show a little life between interactions: while resting, it should quietly notice who's in the room and have a place to hang future reactions — greet a person, perk up at a cat.

Every IDLE_INTERVAL seconds the robot grabs a frame, asks NVIDIA Cosmos Reason 2 who it sees, and routes the observation through a tiny Strands Agent. The whole thing is fully local — local Cosmos vision, local Nemotron routing — so each glance costs $0, in keeping with the idle-is-free principle that runs through the build.

The design choice worth highlighting is the species split. Instead of one generic report_presence, the observation routes to two tools — report_human_presence and report_cat_presence — so people and cats can drive different behaviours. A single glance can fire both (a person and a cat in frame), either alone, or neither.

Goals

• Have the robot periodically notice who's around while idle, fully local and $0
• Use Cosmos Reason 2 to count people and cats and note what each is doing
• Route the observation through a minimal, disposable Strands Agent to species-specific tools
• Support both / either / neither firing per glance — humans and cats are independent
• Never compete with a real task — skip ticks while the robot is busy
• On a real detection, record a clip (S3) and publish a presence state (IoT Core) for downstream systems

The Overall System

A daemon thread ticks on a fixed cadence. Each tick samples the shared camera frame, asks Cosmos a counts-per-species question, and hands the answer to a throwaway agent that routes to the human and/or cat tool — which, on a real detection, record a clip and publish a presence event.

System Components:

1. _idle_watcher — the daemon-thread loop on an IDLE_INTERVAL cadence
2. Shared camera buffer — _latest_jpeg_b64(), the same frame the face tracker maintains
3. Cosmos Reason 2 — answers the counts-per-species IDLE_QUESTION, locally
4. _run_presence_agent — a minimal Strands Agent built per observation and discarded
5. report_human_presence / report_cat_presence — the two species tools, each logging and (on a detection) recording + publishing

Interactive Sequence Diagram

Step through one tick where a person and a cat are in frame — observation, species routing, both tools firing, and the clip/state upload.

One Idle Glance, Routed by Species

A minimal agent sends the observation to human and/or cat tools — both, either, or neither

0/9

Timer

Camera

Cosmos

Agent

Tools

Cloud

Milestone

Complete

9 steps across 6 components • fully local vision + routing, $0 per tick

Idle glances become human / cat events — both, either, or neither

Architecture

Tool	Args	presence_kind	Log colour
report_human_presence	people, description	human	magenta
report_cat_presence	cats, description	cat	cyan

Both are @tool-decorated. Each logs a colour-coded line, and only when its count is >= 1 does it record a short clip (record_clip_and_upload() → S3, presigned URL) and publish a presence state message over IoT Core. The presence agent itself is built and discarded per observation, so nothing persists between ticks.

How it works

The idle watcher

_idle_watcher() runs on a daemon thread started at boot. While the robot is resting it loops on a fixed cadence: every IDLE_INTERVAL seconds (default 10) it grabs the latest frame from the shared camera buffer with _latest_jpeg_b64(), sends it plus IDLE_QUESTION to Cosmos Reason 2, prints the observation, and hands it to a Strands Agent. It is fully local, so each tick costs $0. The default IDLE_QUESTION asks for counts and a short note per species:

How many people and how many cats can you see? Give the count of each (0 if none)
and a short note on what each is doing.

Skipping ticks while a task runs

The watcher checks the shared _busy event at the top of every tick. If a real task is running it logs idle watcher: skip tick (busy with a task) and waits for the next interval, so the presence check never competes with a live look or drives the GPU mid-task:

while True:
    time.sleep(IDLE_INTERVAL)
    if _busy.is_set():
        vlog("idle watcher: skip tick (busy with a task)")
        continue
    ans = _look_via_server(IDLE_QUESTION, image=True, image_b64=_latest_jpeg_b64())
    ...
    _run_presence_agent(ans)

Routing one observation to two tools

_run_presence_agent(observation) builds a minimal Strands Agent for that single observation, registers both presence tools, and discards the agent afterward (del + gc.collect()). Its system prompt routes by species:

• one or more people present → call report_human_presence
• one or more cats present → call report_cat_presence
• both present → call both tools
• neither present → do nothing, emit no prose

The species-specific tools

Each tool logs a line, and only on a real detection does it record and publish — so a fully local run (no S3/MQTT) just logs:

if people >= 1:  # only upload when something is actually detected
    video_url = record_clip_and_upload()
    publish_state("presence", presence_kind="human", presence_count=people,
                  presence_description=description,
                  **({"video_url": video_url} if video_url else {}))

report_cat_presence is identical with presence_kind="cat". Both the clip upload and the state publish are no-ops when S3/MQTT (or the camera owner) are unavailable.

Configuration

Env var	Default	Purpose
IDLE_WATCH	1	Set to 0/false/no/off to disable the watcher
IDLE_INTERVAL	10	Seconds between presence checks
IDLE_QUESTION	counts-per-species prompt	Question sent to Cosmos Reason 2

Technical Challenges & Solutions

Challenge 1: Idle awareness that stays free

Problem: Continuously running vision to watch the room could peg the GPU and, on a cloud VLM, cost money around the clock.

Solution: The watcher ticks on a slow cadence (default every 10s), reuses the already-captured shared frame, and runs local Cosmos + Nemotron — so each glance is $0 and the GPU is touched only briefly, on a timer.

Challenge 2: Presence checks colliding with real tasks

Problem: If a user wakes the robot or an MQTT look arrives mid-tick, two vision workloads would contend for the GPU and camera.

Solution: Every tick first checks the shared _busy event and skips if a task is running. The idle watcher always yields to a real interaction.

Challenge 3: Humans and cats need different reactions

Problem: A single generic report_presence can't let a person and a cat trigger distinct behaviours, and both can be present at once.

Solution: Split into report_human_presence and report_cat_presence, each independently fired by the routing agent. One glance can call both, either, or neither — giving a clean hook for per-species behaviour later, while today each records a clip and publishes a tagged presence event only when its count is real.

Getting Started

GitHub Repository: https://github.com/chiwaichan/nvidia-jetson-thor-strands-agent-reachy-mini-lite

Run it (and tune the cadence)

./reachy_assistant.sh                 # watcher on by default, every 10s
IDLE_INTERVAL=30 ./reachy_assistant.sh # slower cadence
IDLE_WATCH=0 ./reachy_assistant.sh     # disable the watcher entirely

While the robot rests, watch the log: it prints what Cosmos sees each tick and which presence tools fire.

What's Next

In Part 10 - Face Tracking with a Single Camera Owner, the head starts following your face in real time — and I dig into the single camera-owner thread that lets head tracking, the idle watcher, and Cosmos vision all share one camera without V4L2 contention.

Summary

This post covered idle presence awareness:

• Idle glances, fully local — every IDLE_INTERVAL seconds the robot asks Cosmos who it sees, at $0 per tick
• Yields to real work — ticks are skipped while _busy is set, so presence never competes with a live task or the GPU
• Species split — report_human_presence and report_cat_presence route independently; both, either, or neither can fire per glance
• Disposable router — _run_presence_agent builds a minimal agent per observation and dels it, so nothing persists between ticks
• Real detections only — clips (S3) and presence state (IoT Core) are recorded only when a count is >= 1, and no-op when the cloud paths are off