Part of ProjectReachy Mini × Strands Agent on Jetson ThorA local-first voice & vision Reachy Mini Lite robot driven by an on-device Strands Agent on NVIDIA Jetson Thor

This is Part 13 of the series. The telemetry from Part 12 tells a subscriber what the robot did — but not what it saw. This post adds that: each interaction (and each presence detection) is recorded to a short MP4, uploaded to Amazon S3, and a presigned download URL is attached to the outgoing message, so whoever receives the event can watch the exact clip.

The crucial constraint is the camera. The device is owned by a single streaming loop that keeps _latest_frame fresh for every consumer. Opening a second cv2.VideoCapture to record would collide with that owner — so the recorder never does. It samples the shared buffer instead.

The second principle is that video must be invisible when it fails. The entire encode-and-upload path is wrapped so any error is swallowed and the message simply ships without a video_url — a recording problem can never break or slow an interaction.

Goals

• Record each wake interaction (and each presence detection) to a short MP4
• Sample the shared camera buffer — never open a second capture that would fight the owner
• Upload to Amazon S3 and attach a presigned GET URL to the reply / presence message
• Cap memory with a frame budget, and keep encode dependency-light (mp4v, no imageio)
• Swallow all failures — video must never break an interaction — and no-op when the camera or MQTT is off

The Overall System

For a wake, handle_wake starts a recorder thread before the agent runs and stops-and-uploads after the reply. The recorder samples _latest_frame; on stop, the buffered frames are encoded to an MP4, uploaded to S3, and a presigned URL is attached to the reply message.

System Components:

1. start_recording / _record_loop — a daemon thread that samples _latest_frame at VIDEO_FPS
2. Shared camera buffer — _latest_frame, read under _cam_lock
3. _encode_and_upload — cv2.VideoWriter (mp4v) → boto3 upload → presigned URL
4. Amazon S3 — clips land under videos/reachy_<timestamp>.mp4
5. The reply / presence message — carries the video_url when present

Interactive Sequence Diagram

Step through a wake interaction — start recording, sample through the interaction, encode, upload, and attach the presigned URL to the reply.

Recording a Wake Interaction to an S3 Clip

The recorder samples the shared frame buffer — it never opens a second capture

0/9

Wake

Recorder

Buffer

Encode

S3

Reply

Milestone

Complete

9 steps across 6 components • samples the shared buffer, never a second capture

Each interaction becomes an S3 clip with a presigned URL on the reply

Architecture

There are two entry points, both ending in _encode_and_upload:

Path	Trigger	Capture	Duration
start_recording() / stop_recording_and_upload()	wake interaction (handle_wake)	background thread spanning the whole interaction	up to VIDEO_MAX_SECONDS
record_clip_and_upload()	presence detection (report_human_presence / report_cat_presence)	synchronous, inline	VIDEO_CLIP_SECONDS

For a wake, handle_wake calls start_recording() before building the agent and stop_recording_and_upload() after the reply is spoken; stop_recording() runs in finally so the capture thread is always torn down. An instantaneous event like presence has no interaction window, so record_clip_and_upload(seconds) records a fixed-length clip synchronously.

How it works

Sampling the shared camera buffer

Opening a second cv2.VideoCapture would collide with the camera owner, so the recorder samples _latest_frame under _cam_lock and copies each frame:

with _cam_lock:
    frame = _latest_frame
if frame is not None:
    _rec_frames.append(frame.copy())
time.sleep(period)

start_recording() spins a daemon thread (_record_loop) that samples at VIDEO_FPS until stop_recording() sets _rec_stop, or until VIDEO_FPS * VIDEO_MAX_SECONDS frames are buffered (a memory cap). It returns False and is a no-op when there's nowhere to send the result — MQTT off (_iot_conn is None) or the camera owner not streaming (_cam_active not set).

Encoding and uploading to Amazon S3

_encode_and_upload(frames) handles both paths. It writes the buffered frames to a temp file with cv2.VideoWriter using the mp4v fourcc (no extra imageio dependency), uploads with boto3, then returns a presigned URL:

writer = cv2.VideoWriter(tmp_path, cv2.VideoWriter_fourcc(*"mp4v"), VIDEO_FPS, (w, h))
for f in frames:
    writer.write(f)
writer.release()
s3 = boto3.client("s3")
key = f"videos/reachy_{stamp}.mp4"
s3.upload_file(tmp_path, S3_BUCKET, key)
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": S3_BUCKET, "Key": key},
    ExpiresIn=PRESIGNED_URL_EXPIRY,
)

Objects land under videos/ keyed by timestamp. The whole body is wrapped in a try/except that swallows any failure (returning None) and a finally that removes the temp file — encoding or upload must never break an interaction. Empty frame lists short-circuit to None.

Presigned download URLs

The presigned GET URL expires after PRESIGNED_URL_EXPIRY seconds and is attached to the outgoing message as video_url, included only when non-None:

publish_state("reply", request=request, reply=reply,
              **({"video_url": video_url} if video_url else {}))

The same pattern attaches video_url to the presence message.

Configuration

Variable	Default	Purpose
S3_BUCKET	reachy-mini-051826725803	target bucket for uploaded clips
VIDEO_FPS	15	sample rate from the shared buffer and the MP4 frame rate
VIDEO_MAX_SECONDS	120	memory cap on an interaction recording
VIDEO_CLIP_SECONDS	30	length of a one-shot event clip
PRESIGNED_URL_EXPIRY	3600	presigned URL lifetime in seconds

Technical Challenges & Solutions

Challenge 1: Recording without a second camera capture

Problem: The camera is a single-opener device already held by the streaming owner. A recorder that opened its own capture would collide and stall both.

Solution: The recorder samples the shared _latest_frame under _cam_lock and copies each frame — it never touches the device. Recording coexists with face tracking and Cosmos vision because all three read the one owner's buffer.

Challenge 2: Video failures breaking interactions

Problem: Encoding or an S3 upload can fail (disk, network, credentials). If that propagated, a recording problem would break the actual interaction.

Solution: The whole encode-and-upload path is wrapped in try/except that returns None on any failure, with a finally that cleans up the temp file. The reply just ships without a video_url — video is strictly best-effort.

Challenge 3: Unbounded memory on a long interaction

Problem: Buffering frames for an open-ended interaction could grow without limit.

Solution: The recorder caps the buffer at VIDEO_FPS * VIDEO_MAX_SECONDS frames and samples at a modest VIDEO_FPS (default 15), bounding memory while still capturing the interaction.

Challenge 4: Two very different capture windows

Problem: A wake has a natural start/stop window; a presence detection is instantaneous with no window to wrap.

Solution: Two entry points share one encoder — a background thread spans the wake (start_recording / stop_recording_and_upload), while record_clip_and_upload(seconds) grabs a fixed-length clip synchronously for instantaneous events.

Getting Started

GitHub Repository: https://github.com/chiwaichan/nvidia-jetson-thor-strands-agent-reachy-mini-lite

Enable clips

With the camera owner up and MQTT configured, set the bucket and run:

export S3_BUCKET=your-clip-bucket
./reachy_assistant.sh

Each wake reply and presence event that arrives on the MQTT topics now carries a video_url — a presigned link to the clip the robot recorded. When the camera or MQTT is off, both paths are no-ops and messages simply ship without a video_url.

What's Next

In Part 14 - The Camera/Mic Media Bus, I generalise the single-owner idea into a media bus: one broker owns each single-opener device and fans the live stream out over Unix sockets, so any number of processes — including new ones you add — can read the same camera and mic at once.

Summary

This post covered interaction clip recording to S3:

• Sample, don't re-open — the recorder copies the shared _latest_frame under _cam_lock, so it never collides with the camera owner
• Encode + upload — cv2.VideoWriter (mp4v, no imageio) → boto3 upload to videos/reachy_<ts>.mp4 → presigned GET URL
• Best-effort by design — the path swallows all failures (returns None) and is a no-op when the camera or MQTT is off, so video never breaks an interaction
• Two paths, one encoder — a background thread spans a wake; record_clip_and_upload grabs a fixed clip for instantaneous presence events
• Attached to the event — the video_url rides on the reply / presence message, included only when present, expiring after PRESIGNED_URL_EXPIRY