Part of ProjectReachy Mini × Strands Agent on Jetson ThorA local-first voice & vision Reachy Mini Lite robot driven by an on-device Strands Agent on NVIDIA Jetson Thor

This is Part 14 of the series. Several features now want the camera — face tracking, the idle watcher, Cosmos vision, clip recording — and the voice loop wants the mic. But /dev/video0 and the ALSA mic are single-opener devices: once one process holds either, no one else can. Until now everything that needed a device had to live as a thread inside the one process that opened it.

The media bus removes that constraint. One broker process owns each device exactly once and republishes its live stream over a Unix domain socket, so any number of independent processes — the assistant's own loops, and brand-new tools you add later — can read the same frames and audio at once. It's stdlib only — socket, struct, threading, queue — no zmq, no kernel modules.

The design detail that makes it robust is per-subscriber backpressure isolation: each subscriber gets its own bounded queue, and a slow or crashed consumer drops only its own frames, never stalling the device loop or its peers.

Goals

• Let many processes read the single-opener camera and mic at once
• Run one broker per device that owns it exactly once and fans the stream out over a Unix socket
• Force MJPG so the camera runs at ~30 fps instead of the YUYV default's ~5 fps
• Give each subscriber its own bounded queue so backpressure is per-consumer, latest-wins
• Survive slow, late-joining, or crashed consumers without stalling the device loop or peers
• Stay backward compatible — fall back to in-process device ownership when no broker is running

The Overall System

Two broker processes — one for the camera, one for the mic — each own their device and publish length-prefixed messages to every connected subscriber. Subscribers connect over a Unix socket and receive (seq, ts, body) tuples. reachy_assistant.py subscribes through a small client API instead of opening the devices itself.

System Components:

1. media_bus.py camera — owns /dev/video0 (MJPG), publishes JPEG frames
2. media_bus.py audio — owns the mic (arecord), publishes 100 ms S16LE/16k/mono PCM chunks
3. Per-subscriber _Client — a bounded queue.Queue + dedicated sender thread per connection
4. Client API — broker_available, camera_frames, MicReader used by the assistant
5. Subscribers — the face tracker, idle watcher, clip recorder, voice loop, and any new process

Interactive Sequence Diagram

Step through the camera broker fanning one frame out to a fast and a slow subscriber — with a late join and a crashed consumer — and see the survivors keep flowing.

Camera Broker Fan-Out: One Owner, Many Isolated Readers

A slow or crashed subscriber drops only its own frames — the device loop never stalls

0/8

Device

Broker

Sub A

Sub B

Late

Milestone

Complete

8 steps across 5 components • one device owner, per-subscriber backpressure

One camera, fanned out to many — a slow or crashed reader drops only its own frames

Architecture

reachy_assistant.sh starts the two brokers automatically when MEDIA_BUS=1 (the default), each as its own owner process:

python media_bus.py camera   # owns /dev/video0, publishes JPEG frames
python media_bus.py audio    # owns the mic,      publishes S16LE/16k/mono PCM

The camera broker (run_camera_broker) opens the camera once with cv2.CAP_V4L2, captures, resizes to the publish size (640x480 default), JPEG-encodes, and calls broker.publish(...). The audio broker (run_audio_broker) runs one arecord at a fixed format and publishes 100 ms PCM chunks. The format is fixed so subscribers need no negotiation; sockets default to /tmp/reachy_cam.sock and /tmp/reachy_audio.sock.

How it works

The wire format

Every published message is length-prefixed, with a small per-frame header:

[4-byte len][payload]      payload = [seq u64][ts f64][body]

send_msg writes a 4-byte big-endian length then the payload; recv_msg reads the length then exactly that many bytes. The header is seq (a monotonic frame counter set by the broker) and ts (time.time() at publish), followed by the body (JPEG bytes or a PCM chunk). Subscribers receive (seq, ts, body) tuples.

MJPG is essential

OpenCV's VideoCapture defaults to the camera's uncompressed YUYV format, which this camera hard-caps near 5 fps at 1080p. The camera broker forces MJPG via CAP_PROP_FOURCC before reading, lifting the same camera to ~30 fps — the README notes YUYV is hard-capped near 5 fps at 1080p, while MJPG gives ~30+. The earlier ~5 fps was the YUYV default, never a fan-out or hardware limit.

Per-subscriber backpressure isolation

Each subscriber connection becomes a _Client with its own bounded queue.Queue(maxsize=qdepth) (default qdepth=3) and a dedicated sender thread (_send_loop). publish packs the payload once and does a non-blocking put_nowait into every client's queue; if a queue is full, it discards that client's oldest frame and enqueues the new one:

try:
    cl.q.put_nowait(payload)
except queue.Full:
    cl.q.get_nowait()        # drop this client's oldest
    cl.q.put_nowait(payload)
    cl.dropped += 1

Backpressure is therefore strictly per-subscriber and best-effort latest-wins — what a live feed wants. A slow or crashed consumer drops only its own frames and never stalls the device loop or its peers. A dead socket raises in that client's _send_loop, which marks it not alive and closes its connection without touching the others.

How reachy_assistant.py consumes the bus

The assistant imports media_bus and subscribes through a small client API instead of opening the devices itself:

• broker_available(role) — connects to the role's socket to confirm the broker is up
• camera_frames(path=None) — yields decoded BGR numpy frames (cv2.imdecode each JPEG body)
• MicReader(path=None) — a drop-in for the arecord subprocess.Popen the voice loop already uses, exposing .stdout.read(n), .terminate(), .wait(), .kill()

That one camera feed fans out to the face tracker, the idle watcher, and the clip recorder at once — work that previously had to share a single in-process capture. Each consumer is an independent subscription with its own qdepth=3 queue, so they can't stall one another. Because MicReader starts each subscription at live audio, the voice loop's existing "terminate + restart to drop backlog" pattern still works.

The MEDIA_BUS=0 fallback

Because the client API mirrors the existing camera-frame and arecord interfaces, the assistant stays backward compatible. When MEDIA_BUS=0 (or a broker is absent), broker_available(...) returns false and the assistant falls back to opening the device directly and owning it in-process, exactly as before.

Technical Challenges & Solutions

Challenge 1: Single-opener devices, many consumers

Problem: Only one process can hold /dev/video0 or the mic, which forces every camera/mic feature into one process — and blocks any external tool from reading the feed.

Solution: One broker per device owns it exactly once and republishes over a Unix socket. Any number of independent processes subscribe and receive the same (seq, ts, body) stream — the single-opener limit is lifted with stdlib sockets, no extra dependencies.

Challenge 2: A ~5 fps camera

Problem: OpenCV's default YUYV capture caps this camera near 5 fps at 1080p — far too slow for face tracking.

Solution: Force MJPG via CAP_PROP_FOURCC in the broker. The same camera then runs at ~30+ fps at 1080p (vs YUYV's ~5 fps), and because the broker is the only opener, every subscriber benefits from the fast feed.

Challenge 3: A slow or crashed consumer stalling everyone

Problem: In a naive fan-out, one slow reader applies backpressure to the shared stream and stalls the device loop and every other consumer; a crashed reader could wedge the broker.

Solution: Each subscriber gets its own bounded queue and sender thread. A full queue drops that subscriber's oldest frame (latest-wins); a dead socket closes only that connection. The device loop and surviving subscribers are never affected — survivors see 0 drops.

Challenge 4: Not breaking the standalone run

Problem: Adding a broker shouldn't make the assistant depend on it.

Solution: The client API mirrors the in-process camera-frame and arecord interfaces, so with MEDIA_BUS=0 or no broker present, the assistant transparently falls back to owning the devices itself — identical behaviour to before the bus existed.

Getting Started

GitHub Repository: https://github.com/chiwaichan/nvidia-jetson-thor-strands-agent-reachy-mini-lite

Run with or without the bus

./reachy_assistant.sh                 # starts both brokers, then the assistant (MEDIA_BUS=1)
MEDIA_BUS=0 ./reachy_assistant.sh     # assistant owns the camera + mic in-process, as before

# or run a broker by hand
python media_bus.py camera            # owns /dev/video0, publishes JPEG frames
python media_bus.py audio             # owns the mic, publishes PCM chunks

With the brokers up, the face tracker, idle watcher, clip recorder, and voice loop all read the same camera and mic — and you can attach a brand-new subscriber without disturbing any of them.

What's Next

In Part 15 - Conversational Memory Across Wakes, the robot starts remembering recent turns — ask a follow-up across separate wakes and the fresh per-wake agent recalls it — built on Strands' session managers, stored as local JSON, reboot-safe, and still $0 idle.

Summary

This post covered the camera/mic media bus:

• One owner per device — media_bus.py runs a broker that opens the camera or mic exactly once and fans the stream out over a Unix socket, stdlib only
• MJPG for ~30 fps — forcing the FOURCC lifts the camera from ~5 fps (YUYV) to ~30+ fps at 1080p
• Length-prefixed wire format — [4-byte len][seq u64][ts f64][body], delivered to subscribers as (seq, ts, body)
• Per-subscriber backpressure — each connection has its own bounded queue + sender thread; a slow/crashed consumer drops only its own frames, latest-wins, never stalling the device loop or peers
• Drop-in + backward compatible — camera_frames and MicReader mirror the existing interfaces, and MEDIA_BUS=0 falls back to in-process ownership