I got me one of these!

This thing is a beast and I am keen to see what kind of AI robotics experiments I can create from this.

These days I am often creating small generic re-usable building blocks that I can pontentially use across new or existing projects, in this blog I talk about the architecture for a LLM based voice chatbot in a web browser built entirely as a serverless based solution.

The key component of this solution is using Amazon Nova 2 Sonic, a speech-to-speech foundation model that can understand spoken audio directly and generate voice responses - all through a single bidirectional stream from the browser directly to Amazon Bedrock, with no backend servers required - no EC2 instances and no Containers.

Goals

• Enable real-time voice-to-voice conversations with AI using Amazon Nova 2 Sonic
• Direct browser-to-Bedrock communication using bidirectional streaming - no Lambda functions or API Gateway required
• Use AWS Amplify Gen 2 for infrastructure-as-code backend definition in TypeScript
• Implement secure authentication using Cognito User Pool and Identity Pool for temporary AWS credentials
• Handle real-time audio capture, processing, and playback entirely in the browser
• Must be a completely serverless solution with automatic scaling
• Support click-to-talk interaction model for intuitive user experience
• Display live transcripts of both user speech and AI responses

Architecture

End-to-End Voice Chat Flow​

This diagram illustrates the complete flow from a user speaking into their microphone to hearing the AI assistant's voice response.

Flow Steps:

1. User Speaks - User clicks the microphone button and speaks naturally
2. Audio Capture - Browser captures audio via Web Audio API at 48kHz
3. Authentication - React app authenticates with Cognito User Pool
4. Token Exchange - JWT tokens exchanged for Identity Pool credentials
5. AWS Credentials - Temporary AWS credentials (access key, secret, session token) returned
6. Bidirectional Stream - Audio streamed to Bedrock via InvokeModelWithBidirectionalStream
7. Voice Response - Nova Sonic processes speech and returns synthesized voice response
8. Audio Playback - Response audio decoded and played through speakers
9. User Hears - User hears the AI assistant's natural voice response

Interactive Sequence Diagram​

React Hooks Architecture​

This diagram details the internal architecture of the React application, showing how custom hooks orchestrate audio capture, Bedrock communication, and playback.

Components:

• VoiceChat.tsx - Main UI component that coordinates all hooks and renders the interface
• useNovaSonic - Core hook managing Bedrock bidirectional stream, authentication, and event protocol
• useAudioRecorder - Captures microphone input using AudioWorklet in a separate thread
• useAudioPlayer - Manages audio playback queue and Web Audio API buffer sources
• audioUtils.ts - Low-level utilities for PCM conversion, resampling, and Base64 encoding

Data Flow:

1. Microphone audio captured by useAudioRecorder via MediaStream
2. AudioWorklet processes samples in real-time (separate thread)
3. Audio resampled from 48kHz to 16kHz, converted to PCM16, then Base64
4. useNovaSonic streams audio chunks to Bedrock
5. Response audio received as Base64, decoded to PCM, converted to Float32
6. useAudioPlayer queues AudioBuffers and plays through AudioContext

Authentication Flow​

This diagram shows the multi-layer authentication flow that enables secure browser-to-Bedrock communication without exposing long-term credentials.

Authentication Layers:

• Cognito User Pool - Handles user registration and login with email/password
• Cognito Identity Pool - Exchanges JWT tokens for temporary AWS credentials
• IAM Role - Defines permissions for authenticated users (Bedrock invoke access)
• SigV4 Signing - AWS SDK automatically signs all Bedrock requests

Key Security Features:

• No AWS credentials stored in browser - only temporary session credentials
• Credentials automatically refreshed by Amplify SDK before expiration
• IAM policy scoped to specific Bedrock model (amazon.nova-2-sonic-v1:0)
• All communication over HTTPS with TLS 1.2+

Audio Processing Pipeline​

This diagram shows the real-time audio processing that converts browser audio to Bedrock's required format and vice versa.

Input Processing (Recording):

1. Microphone - Browser captures audio at native sample rate (typically 48kHz)
2. AudioWorklet - Processes audio in separate thread, accumulates 2048 samples
3. Resample - Linear interpolation converts 48kHz → 16kHz (Nova Sonic requirement)
4. Float32 → PCM16 - Converts floating point [-1,1] to 16-bit signed integers
5. Base64 Encode - Binary PCM encoded for JSON transmission

Output Processing (Playback):

1. Base64 Decode - Received audio converted from Base64 to binary
2. PCM16 → Float32 - 16-bit integers converted to floating point
3. AudioBuffer - Web Audio API buffer created at 24kHz (Nova Sonic output rate)
4. Queue & Play - Buffers queued and played sequentially through speakers

Interactive Sequence Diagram​

Bidirectional Streaming Protocol​

This diagram illustrates how the useNovaSonic hook manages the complex bidirectional streaming protocol with Amazon Bedrock.

Event Protocol: Nova Sonic uses an event-based protocol where each interaction consists of named sessions, prompts, and content blocks.

Input Events (sent to Bedrock):

• sessionStart - Initializes session with inference parameters (maxTokens: 1024, topP: 0.9, temperature: 0.7)
• promptStart - Defines output audio format (24kHz, LPCM, voice "matthew")
• contentStart - Marks beginning of content blocks (TEXT for system prompt, AUDIO for user speech)
• textInput - Sends system prompt text content
• audioInput - Streams user audio chunks as Base64-encoded 16kHz PCM
• contentEnd - Marks end of content block
• promptEnd / sessionEnd - Terminates prompt and session

Output Events (received from Bedrock):

• contentStart - Marks role transitions (USER for ASR, ASSISTANT for response)
• textOutput - Returns transcribed user speech and generated AI response text
• audioOutput - Returns synthesized voice response as Base64-encoded 24kHz PCM
• contentEnd - Marks end of response content

Async Generator Pattern: The SDK requires input as AsyncIterable<Uint8Array>. The hook implements this using:

• Event Queue - Pre-queued initialization events before stream starts
• Promise Resolver - Backpressure control for yielding events on demand
• pushEvent() - Adds new events during conversation (audio chunks)

Serverless Architecture Overview​

This diagram provides a comprehensive view of all components - the entire solution is serverless with no EC2 instances or containers to manage.

Frontend Stack:

• React - Component-based UI framework
• Vite - Build tool and dev server
• TypeScript - Type-safe development
• AWS Amplify Hosting - Static web hosting with global CDN

Backend Stack (Amplify Gen 2):

• amplify/backend.ts - Infrastructure defined in TypeScript
• Cognito User Pool - Email-based authentication
• Cognito Identity Pool - AWS credential vending
• IAM Policy - Grants bedrock:InvokeModel permission for bidirectional streaming

AI Service:

• Amazon Bedrock - Managed foundation model inference
• Nova 2 Sonic - Speech-to-speech model (us-east-1)
• Bidirectional Streaming - Real-time duplex communication

Technical Challenges & Solutions

Challenge 1: AudioWorklet CORS Issues​

Problem: Loading AudioWorklet from external file fails with CORS errors on some deployments.

Solution: Inline the AudioWorklet code as a Blob URL:

const blob = new Blob([audioWorkletCode], { type: 'application/javascript' });
const workletUrl = URL.createObjectURL(blob);
await audioContext.audioWorklet.addModule(workletUrl);
URL.revokeObjectURL(workletUrl);

Challenge 2: Sample Rate Mismatch​

Problem: Browsers capture audio at 48kHz, but Nova Sonic requires 16kHz input.

Solution: Linear interpolation resampling in real-time:

const resampleAudio = (audioData: Float32Array, sourceSampleRate: number, targetSampleRate: number) => {
  const ratio = sourceSampleRate / targetSampleRate;
  const newLength = Math.floor(audioData.length / ratio);
  const result = new Float32Array(newLength);
  for (let i = 0; i < newLength; i++) {
    const srcIndex = i * ratio;
    const floor = Math.floor(srcIndex);
    const ceil = Math.min(floor + 1, audioData.length - 1);
    const t = srcIndex - floor;
    result[i] = audioData[floor] * (1 - t) + audioData[ceil] * t;
  }
  return result;
};

Challenge 3: SDK Bidirectional Stream Input​

Problem: AWS SDK requires input as AsyncIterable<Uint8Array>, but events need to be pushed dynamically during the conversation.

Solution: Async generator with event queue and promise-based backpressure:

async function* createInputStream() {
  while (isActiveRef.current && !ctrl.closed) {
    while (ctrl.eventQueue.length > 0) {
      yield ctrl.eventQueue.shift();
    }
    const nextEvent = await new Promise(resolve => {
      ctrl.resolver = resolve;
    });
    if (nextEvent === null) break;
    yield nextEvent;
  }
}

Getting Started

GitHub Repository: https://github.com/chiwaichan/amplify-react-amazon-nova-2-sonic-voice-chat

Prerequisites​

• Node.js 18+
• AWS Account with Bedrock access enabled
• AWS CLI configured with credentials

Deployment Steps​

1. Enable Nova 2 Sonic in Bedrock Console (us-east-1 region)

2. Clone and Install:

git clone https://github.com/chiwaichan/amplify-react-amazon-nova-2-sonic-voice-chat.git
cd amplify-react-amazon-nova-2-sonic-voice-chat
npm install

3. Start Amplify Sandbox:

npx ampx sandbox

4. Run Development Server:

npm run dev

5. Open Application: Navigate to http://localhost:5173, create an account, and start talking!

Summary

This architecture provides a reusable building block for voice-enabled AI applications:

• Zero backend servers - Direct browser-to-Bedrock communication
• Real-time streaming - HTTP/2 bidirectional streaming for low latency
• Secure authentication - Cognito User Pool + Identity Pool + IAM policies
• Audio processing pipeline - Web Audio API, AudioWorklet, PCM conversion
• Infrastructure as code - AWS Amplify Gen 2 with TypeScript backend definition

The entire interaction happens in real-time: speak naturally, and hear the AI respond within seconds.

A Sphero RVR integrated with a Seeed Studio XIAO ESP32S3 with telemetry uploaded into, and also, basic drive remote control commands received from any where leveraging AWS IoT Core.

Overview​

Lately I have been aiming to go deep on AI Robotics, and last year I have been slowly experimenting more and more with anything that is AI, IoT and Robotics related; with the intention of learning and going as wide and as deep as possible in any pillars I can think of. You can check out my blogs under the Robotics](/projects/robotics) Project to see what I have been up to. This year I want to focus on enabling mobility for my experiments - as in providing wheels for solutions to move around the house, ideally autonomously; starting off with wheel based solutions bought off-shelve, followed by solutions that I build myself from open-sourced projects people have kindly contirbuted online, and then ambitiously designed, 3D Printed and built all from the ground up - perhaps in a couple of years time.

This project uses a Seeed Studio XIAO ESP32S3 microcontroller to communicate with a Sphero RVR robot via UART, while simultaneously connecting to AWS IoT Core over WiFi. The system publishes real-time sensor telemetry and accepts remote drive commands through MQTT.

Hardware Components​

Hardware Wiring​

Features​

Real-time Telemetry​

The system publishes comprehensive sensor data every 60 seconds:

• IMU Data: Pitch, roll, yaw orientation
• Accelerometer & Gyroscope: Motion and rotation data
• Color Sensor: RGB values with confidence
• Compass: Heading in degrees
• Ambient Light: Lux measurements
• Motor Thermal: Temperature and protection status
• Encoders: Wheel tick counts
• Position & Velocity: Locator data in meters

Remote Commands via MQTT​

Control the RVR from anywhere using JSON commands:

• Drive: Speed and heading control
• Tank: Independent left/right motor control
• Raw Motors: Direct motor speed control
• LED Control: Headlights, brakelights, status LEDs
• Navigation: Reset yaw, reset locator
• Power: Wake and sleep commands

Local OLED Display​

The XIAO Expansion Board's OLED display shows real-time sensor readings for local monitoring.

MQTT Message Flow​

Sensor Data Pipeline​

Architecture​

The XIAO ESP32S3 acts as a bridge between the Sphero RVR and AWS IoT Core:

1. UART Communication: The ESP32S3 communicates with the RVR via UART (GPIO43/44)
2. WiFi Connection: Connects to local WiFi network
3. MQTT over TLS: Secure connection to AWS IoT Core with X.509 certificates
4. Bidirectional: Publishes telemetry and subscribes to command topics

Sphero RVR Protocol​

The Sphero RVR uses a binary packet-based protocol over UART. Each packet contains a start-of-packet byte (0x8D), an 8-byte header with device ID and command ID, variable-length data body, checksum, and end-of-packet byte (0xD8). The RVR has two internal processors: Nordic (handles BLE, power, color detection) and ST (handles motors, IMU, encoders).

Source Code​

I ported the code into this project to control the RVR using the UART protocol based on the Sphero SDK.

You can find the source code for this project here: https://github.com/chiwaichan/platformio-aws-iot-seeed-studio-esp32s3-sphero-rvr

The LeRobot Follower arm is subscribed to an IoT Topic that is being published in real-time by the LeRobot Leader arm over AWS IoT Core, using a Seeed Studio XIAO ESP32C3 integrated with a Seeed Studio Bus Servo Driver Board, the driver board is controlling the 6 Feetech 3215 Servos over the UART protocol.

In this video I demonstrate how to control a set of Hugging Face SO-101 arms over AWS IoT Core, without the use of the LeRobot framework, nor using a device such as a Mac nor a device like Nvidia Jetson Orin Nano Super Developer Kit. Only using Seeed Studio XIAO ESP32C3 and AWS IoT.

You can find the source code for this solution here: https://github.com/chiwaichan/aws-iot-core-lerobot-so101

Good times

I was hungry and tried to feed myself but I forgot to bring cat food.

Here is the source-code Streamlit App I demoed at the AWS Sydney Summit, to demonstrate how to control IoT devices using the Strands Agents: https://github.com/chiwaichan/AWSSydneySummit2025Demo