Skip to main content

Voice to Robotics: Driving a Sphero RVR — AWS User Group Wellington

· 3 min read
Chiwai Chan
Chiwai Chan
Tinkerer

These are the slides from my talk at the AWS User Group Wellington meet-up on 26 May 2026, walking through how I built a real-time pipeline that listens to spoken commands in the browser and drives a Sphero RVR — exact distances, exact angles — with live telemetry streaming back into the UI.

Tip: click into the slides and use the arrow keys to navigate, or hit the fullscreen button for the best experience.

What the talk covers​

  • The problem — turning a spoken sentence into a precise, exact-distance robot motion in real time
  • Voice in — Amazon Nova 2 Sonic bidirectional streaming with forced tool use as a "robot controller"
  • Commands down — browser → AWS IoT Core MQTT publish via temporary Cognito credentials, no backend
  • The edge — a Seeed Studio XIAO ESP32-S3 bridging WiFi/MQTT to the Sphero RVR's UART SDK
  • Maneuver framework — locator-based distance and IMU-based turn, both running on the device, so "1 metre" actually means 1 metre
  • Telemetry up — an AWS CDK stack tying IoT Rule → Lambda → AppSync → DynamoDB → live GraphQL subscription back into the React UI

The three repos behind the demo​

  • arduino-aws-iot — ESP32-S3 firmware. The talk focuses on the sphero_rvr device group, including the new maneuver executor (forward, reverse, turn, cancel) layered on top of the existing drive D-pad.
  • cdk-iot-sphero-rvr-streaming — AWS CDK stack that filters the RVR telemetry MQTT topic, flattens the nested payload in Lambda, and forwards it as a GraphQL mutation to an Amplify-managed AppSync API.
  • amplify-react-nova-sonic-voice-chat-sphero-rvr — React + AWS Amplify Gen 2 frontend. The same Cognito identity is used for Bedrock streaming, IoT publish, and the AppSync subscription that surfaces live telemetry.

If you want background on the building blocks before the talk:

  1. Sphero RVR with AWS IoT Core on a Seeed Studio XIAO ESP32-S3
  2. Real-Time Voice Chat with Amazon Nova Sonic, React and AWS Amplify Gen 2

Thanks​

Big thanks to the AWS User Group Wellington organisers and everyone who came along — happy to chat about any of the code, the maneuver framework, or where the project goes next.