Skip to main content
SYS.ONLINE

Reachy Mini x Strands Agent on Jetson Thor - Part 10: Real-Time Face Tracking with a Single Camera Owner

· 9 min read
Chiwai Chan
Chiwai Chan
Tinkerer
Part of ProjectReachy Mini × Strands Agent on Jetson ThorA local-first voice & vision Reachy Mini Lite robot driven by an on-device Strands Agent on NVIDIA Jetson Thor

A single camera-owner thread publishes the latest frame to the face tracker and Cosmos Reason 2

This is Part 10 of the series. The robot can already look on demand with Cosmos vision and notice who's around while idle. Now I want it to feel present: its head should follow your face in real time — and it has to do that while Cosmos vision and the idle watcher are also using the camera.

Tracking itself is simple and entirely local — an OpenCV Haar cascade finds the nearest face, and a proportional controller nudges the head to centre it, offline and $0. The hard part isn't the tracking; it's the camera. /dev/video0 is a single-opener V4L2 device, so the face tracker, the idle watcher, the clip recorder, and the Cosmos look path can't each open their own capture.

The answer is a single camera-owner thread: one reader of the device that publishes the latest frame to a shared buffer, and every other consumer reads that. Head tracking and Cosmos vision run off the same frames at the same time, with no contention.

Goals

  • Make the head follow the nearest face in real time with local OpenCV — offline, $0
  • Run a single camera-owner thread as the only reader of the device, publishing frames to a shared buffer
  • Let the tracker, idle watcher, and Cosmos vision all read the same frame — no V4L2 contention
  • Drive the head with a proportional controller with a deadband (anti-jitter) and mechanical clamps
  • Yield the head to a real task: pause and recenter while the robot is busy
  • Make tracking tunable and easy to disable via FACE_* env vars

The Overall System

One thread owns the camera and keeps a shared frame fresh. The tracker reads it, finds the largest face, computes a centring error, and steps the head. The same buffer feeds Cosmos when the agent calls look_and_describe, so vision never opens a second capture.

System Components:

  1. _capture_loop (camera-owner thread) — the only device reader; keeps _latest_frame fresh under _cam_lock
  2. Shared frame buffer — _latest_frame (BGR) and _latest_jpeg_b64() for consumers
  3. _tracker_loop (face-tracker thread) — Haar detection + proportional control of the head
  4. Reachy head — driven via goto_target at ~10 Hz
  5. Cosmos look path — reads the same shared frame as image_b64, never the device

Interactive Sequence Diagram​

Step through one ~10 Hz tracking cycle — capture, detect, control, move — and see Cosmos share the same frame on a look.

One Tracking Cycle: Frame to Head Move at ~10 Hz

A single camera owner feeds the tracker and Cosmos from the same frame

0/7
OwnerCamera OwnerBufferFrame BufferTrackerFace TrackerControlP-ControllerHeadReachy HeadCosmosCosmos (vision)capturepublish latest BGR frame (MJPG @ 1920x1080)sole reader of /dev/video0readcopy _latest_frame; resize 320x240, gray_busy? recenter & skipdetectdetectMultiScale (Haar) -> largest facenearest = largest areaerrornormalized error ex, ey in [-1, 1] vs centremovegain*err, clamp (yaw 55, pitch 22) -> goto_targetdeadband + FACE_MOVE_DUR smoothingsharesame frame as image_b64 on a lookno second capture, no V4L2 contentionloopsleep FACE_MOVE_PERIOD; repeat (~10 Hz)pauses & recenters while busy
Owner
Buffer
Tracker
Control
Head
Cosmos
Milestone
Complete
7 steps across 6 components • one camera owner, many readers
Head follows the nearest face at ~10 Hz — Cosmos shares the same frame

Architecture

ThreadRole
_capture_loop (camera-owner)Sole reader of /dev/video0 (or the media-bus broker); publishes _latest_frame
_tracker_loop (face-tracker)Haar detection, error computation, proportional head control

_capture_loop starts whenever FACE_TRACK is on or the camera broker is active; _tracker_loop starts when FACE_TRACK is on. Every in-process consumer reads the shared buffer rather than opening its own cv2.VideoCapture.

How it works

One camera owner, many readers​

_capture_loop() runs as the camera-owner daemon thread and keeps _latest_frame (a BGR numpy array) fresh under _cam_lock. When the media-bus camera broker is up it subscribes to that; otherwise it opens /dev/video<CAMERA> directly with V4L2 and MJPG at 1920x1080. Once streaming, it sets the _cam_active event. Every other in-process consumer reads that shared buffer — so the face tracker and the Cosmos look path run off the same frames at the same time, with no V4L2 contention.

Detecting the largest face​

_tracker_loop() waits up to ten seconds for _cam_active, then loads the Haar cascade haarcascade_frontalface_default.xml. Each cycle it copies _latest_frame under _cam_lock, resizes to 320x240, converts to gray, and runs detectMultiScale(gray, 1.1, 4, minSize=(30, 30)). It picks the largest face by area as the nearest, and normalises its offset from the frame centre to an error in [-1, 1]:

x, y, w, h = max(faces, key=lambda f: f[2] * f[3])   # largest = nearest face
ex = ((x + w / 2) - 160) / 160.0                      # horizontal error [-1, 1]
ey = ((y + h / 2) - 120) / 120.0                      # vertical error   [-1, 1]

Proportional controller with deadband and clamps​

The errors drive the head. Errors smaller than FACE_DEADBAND are ignored to suppress jitter; otherwise each axis steps by gain × error × sign and clamps to the head's mechanical limits (FACE_YAW_MAX 55°, FACE_PITCH_MAX 22°):

if abs(ex) > FACE_DEADBAND:
    tyaw = _clamp(tyaw + FACE_KP_YAW * ex * FACE_YAW_SIGN, -FACE_YAW_MAX, FACE_YAW_MAX)
if abs(ey) > FACE_DEADBAND:
    tpitch = _clamp(tpitch + FACE_KP_PITCH * ey * FACE_PITCH_SIGN, -FACE_PITCH_MAX, FACE_PITCH_MAX)

When either axis moved, the loop calls mini.goto_target(create_head_pose(yaw=tyaw, pitch=tpitch, degrees=True), duration=FACE_MOVE_DUR) and sleeps FACE_MOVE_PERIOD (0.1 s, ~10 Hz) per cycle.

Pausing and recentering while a task owns the head​

The tracker yields the head whenever a task is running. At the top of each cycle it checks the _busy event; if a task owns the head it resets the target to tyaw = tpitch = 0.0 and skips the rest, so tracking resumes from centre once the task finishes. The Cosmos look path stays available throughout because it only reads the shared frame, never the motors.

The FACE_* knobs​

Env varDefaultEffect
FACE_TRACK1Enable face tracking (0 disables; Cosmos server self-captures)
FACE_KP_YAW20Yaw gain, degrees per step per unit error
FACE_KP_PITCH16Pitch gain, degrees per step per unit error
FACE_DEADBAND0.06Ignore errors smaller than this (anti-jitter)
FACE_MOVE_PERIOD0.1Head-update period in seconds (~10 Hz)
FACE_MOVE_DUR0.12Smoothing duration per head move
FACE_YAW_SIGN / FACE_PITCH_SIGN-1 / 1Flip if the head moves the wrong way

FACE_YAW_MAX (55) and FACE_PITCH_MAX (22) are fixed in code as the clamp limits.

Technical Challenges & Solutions

Challenge 1: A single-opener camera, many consumers​

Problem: /dev/video0 is a single-opener V4L2 device. The face tracker, idle watcher, clip recorder, and Cosmos look path all need frames — but only one process can hold the device.

Solution: A single camera-owner thread is the only reader; it publishes _latest_frame to a lock-guarded shared buffer (and _latest_jpeg_b64() for the VLM). Every consumer reads the buffer, so head tracking and vision coexist with zero contention.

Challenge 2: Jittery, twitchy head motion​

Problem: A raw proportional controller chases every tiny detection wobble, producing constant micro-movements that look nervous.

Solution: A deadband (FACE_DEADBAND) ignores small errors, the per-move smoothing duration (FACE_MOVE_DUR) eases each step, and the ~10 Hz update period keeps motion controlled. Gains and signs are env-tunable so the response can be dialled in per robot.

Challenge 3: Tracking fighting a real task for the head​

Problem: If the tracker kept driving the head while a wake task tried to nod or look around, the two would fight over the motors.

Solution: The tracker checks the shared _busy event each cycle and, while a task owns the head, resets its target to centre and skips. It resumes from neutral once the task ends — and Cosmos vision is unaffected throughout because it only reads frames.

Getting Started

GitHub Repository: https://github.com/chiwaichan/nvidia-jetson-thor-strands-agent-reachy-mini-lite

Run it (and tune the tracking)​

./reachy_assistant.sh                  # face tracking on by default
FACE_TRACK=0 ./reachy_assistant.sh     # disable; Cosmos server self-captures
FACE_YAW_SIGN=1 ./reachy_assistant.sh  # flip if the head turns the wrong way

Stand in front of the robot and move side to side — the head follows. Say "Hey Reachy" and the head recenters for the task, then resumes tracking afterward.

What's Next

In Part 11 - Going Fully Local with Nemotron via Ollama, I swap the agent's brain from Amazon Bedrock to a local Nemotron model served by Ollama on the Thor GPU — so the whole assistant, brain included, runs on-device at $0.

Summary

This post covered real-time face tracking and the single-camera-owner design:

  • Head follows the nearest face — an OpenCV Haar cascade picks the largest face and a proportional controller centres it, locally and $0
  • One camera owner, many readers — a single camera-owner thread is the sole device reader; the tracker, idle watcher, and Cosmos all read the shared _latest_frame, so there's no V4L2 contention
  • Smooth control — a deadband suppresses jitter, gains/signs are env-tunable, and the head is clamped to its mechanical limits (yaw 55°, pitch 22°) at ~10 Hz
  • Yields to tasks — the tracker recenters and pauses while _busy is set, then resumes, while Cosmos vision keeps working off the shared frame
  • Frame sharing — look_and_describe passes the owner's latest frame as image_b64, so vision never opens a second capture