Skip to main content
SYS.ONLINE

Voice to Sign Language Translation

The black 3D-printed Pollen Robotics Amazing Hand robotic hand on a stand, fingers raised mid-gesture
The Pollen Robotics Amazing Hand — an open-source robotic hand that fingerspells the ASL alphabet.

Overview​

This project is a real-time voice-to-sign-language translation system that takes spoken words from a browser microphone and translates them into physical ASL fingerspelling on a Pollen Robotics Amazing Hand. The system spans three repositories covering the frontend, cloud infrastructure, and edge AI agent.

End-to-end system architecture: browser microphone to a React frontend, Amazon Nova 2 Sonic for speech, Cognito auth, MQTT over AWS IoT Core routed through Lambda to AppSync, and a Strands Agent on NVIDIA Jetson driving the Amazing Hand servos, with a Three.js 3D hand synced via GraphQL
The end-to-end flow — browser speech → Nova 2 Sonic → MQTT → a Strands Agent on NVIDIA Jetson drives the Amazing Hand, with a 3D hand synced back via GraphQL.

Demo​

Technologies Used​

  • Frontend: React 19, Vite 7, TypeScript, Three.js, AWS Amplify Gen 2
  • Voice AI: Amazon Nova 2 Sonic (speech-to-speech, bidirectional streaming)
  • Edge AI: Strands Agents framework, Amazon Nova 2 Lite (tool-use reasoning)
  • Cloud: AWS IoT Core, AWS AppSync, AWS Lambda, Amazon DynamoDB, Amazon S3
  • Hardware: NVIDIA Jetson, Pollen Robotics Amazing Hand, Feetech SCS0009 servos
  • Infrastructure: AWS CDK, GitHub Actions CI/CD
  • Protocols: MQTT, GraphQL subscriptions, Serial (1M baud)
The React web app's three-column UI: signed-letter history, a 3D robotic hand visualisation with a video feed, and a live transcript with microphone controls
The React frontend — a live transcript, a Three.js 3D hand synced to the physical hand, and the signed-letter history.

Key Features​

  • Direct browser-to-Bedrock bidirectional streaming with Nova 2 Sonic
  • Forced tool use (send_text) to relay cleaned speech as text on every utterance
  • ASL fingerspelling with 26 alphabet letters (A-Z)
  • Real-time 3D hand visualisation synchronised with the physical hand via GraphQL subscriptions
  • Video recording with H.264 encoding, S3 upload, and presigned URLs
  • Fully serverless frontend with Cognito authentication
The edge AI agent architecture: an MQTT listener on the NVIDIA Jetson feeds a Strands Agent powered by Amazon Nova 2 Lite, which invokes a fingerspell tool to drive the Amazing Hand's Feetech servos and records video to S3
The edge AI agent — a Strands Agent (Amazon Nova 2 Lite) on the Jetson turns MQTT sentence text into ASL servo commands, letter by letter.

GitHub Repositories

amplify-react-nova-sonic-voice-chat-amazing-hand

React frontend with Nova 2 Sonic voice processing and MQTT servo command publishing

View on GitHub →

cdk-iot-amazing-hand-streaming

AWS CDK stack routing IoT Core MQTT messages through Lambda to AppSync

View on GitHub →

strands-agents-amazing-hands

Strands Agent on NVIDIA Jetson driving Amazing Hand servos for ASL fingerspelling

View on GitHub →