Skip to main content

Voice to Robotics: Fingerspelling American Sign Language — AWS User Group Wellington

· 2 min read
Chiwai Chan
Chiwai Chan
Tinkerer

These are the slides from my talk at the AWS User Group Wellington meet-up on 29 April 2026, walking through how I built a real-time pipeline that listens to speech in the browser and drives a robotic hand to fingerspell the words in American Sign Language (ASL).

Tip: click into the slides and use the arrow keys to navigate, or hit the fullscreen button for the best experience.

What the talk covers​

  • The problem — bridging spoken language and ASL fingerspelling in real time, end-to-end
  • Voice in — Amazon Nova 2 Sonic bidirectional streaming as a "dumb" speech-to-text relay with forced tool use
  • Cloud glue — AWS IoT Core MQTT, AppSync subscriptions, and an AWS CDK stack tying it all together
  • Edge AI agent — a Strands Agent on an NVIDIA Jetson translating sentences into servo commands
  • The hand — driving the Pollen Robotics Amazing Hand for ASL fingerspelling
  • Lessons learned — latency, reliability, and what I'd do differently

If you want to go deeper than the slides allow, the full three-part write-up lives here:

  1. Part 1 — Frontend and Voice Processing
  2. Part 2 — Cloud Infrastructure (IoT, AppSync, CDK)
  3. Part 3 — Edge AI Agent (Strands, NVIDIA Jetson, Amazing Hand)

Thanks​

Big thanks to the AWS User Group organisers and everyone who came along — happy to chat about any of this, the code, or where it goes next.