Introducing Speakrail: A Fully-Local Voice Agent That Competes with GPT-Live on an RTX 4090
In the rapidly evolving field of AI-driven voice assistants, achieving real-time, high-quality interactions while maintaining user privacy has been a persistent challenge. Today, I am excited to share a breakthrough: Speakrail, an open-source, fully-local speech interaction system capable of rivaling commercial solutions like GPT-Live—all powered by a single NVIDIA RTX 4090.
Why Develop Speakrail?
The inspiration behind Speakrail stems from a desire for privacy-conscious, low-latency voice assistants. Traditional pipelines often rely on cloud components, raising privacy concerns and introducing latency. While efforts like Hugging Face’s speech-to-speech models, Unmute, and Pipecat have explored fully local solutions, they encountered limitations—either due to model hallucinations from Whisper or slow turn-taking speeds.
A notable previous approach, DuplexCascade, utilized a Qwen 2 7B model trained on GPT-generated datasets, but it struggled with dialogue complexity and responsiveness. I envisioned a more robust solution that combines modern models, efficient architecture, and advanced turn-taking capabilities to create a truly conversational, local voice agent.
How Does Speakrail Work?
Speakrail integrates multiple state-of-the-art components into a seamless pipeline:
-
Speech-to-Text (STT): Utilizing Voxtral Realtime with a custom turn-taking head, running efficiently via audio.cpp. This setup ensures rapid, accurate transcription with natural turn management.
-
Large Language Model (LLM): Powered by Gemma 4 12B QAT, micro-tuned for conversational reasoning and efficiency. Gemma’s versatility and performance in real-world tasks make it an excellent choice for embedded setups.
-
Text-to-Speech (TTS): Employing Breeze TTS 2, modified to run on int8 quantization, delivering expressive, streaming speech output. Its architecture supports real-time TTS, but it can be swapped for other streaming TTS models as needed.
-
Middleware & Management: The “glue” that orchestrates all components—implemented with latency optimizations such as speculative firing of LLM and TTS modules, inspired by speech-to-speech systems like Hugging Face’s. Additionally, it incorporates innovative “think while talking” features, where the system processes reasoning notes in the background, enabling more complex interactions.
Performance Benchmarks
In rigorous testing, Speakrail demonstrates impressive capabilities:
| System | Pass@1 | Tool F1 | Reply Quality | Task Completion Time | Status |
|————|———|———|————–|——————|———|
| TML-Interaction-Small | 0.68 | – | 0.828 | – | Preview |
| GPT-Realtime 1.5 | 0.60 | 0.876 | 0.792 | 6.89s | Closed |
| GPT-Live + Astra | 0.55 | 0.835 | 0.717 | 8.47s | Closed |
| Gemini Live 3.1 | 0.54 | 0.817 | 0.718 | 4.25s | Closed |
| Speakrail (Our System) | 0.52 | 0.915 | 0.755 | 3.92s | Open Source |
Remarkably, Speakrail’s reply quality approaches that of GPT-Live with Astra, while outperforming others in task speed and tool selection efficiency.
In estimating conversational dynamics, our analysis indicates Speakrail maintains a high comprehension score comparable to leading systems, despite running on a modest GPU.
Limitations & Future Improvements
While promising, certain aspects warrant further refinement:
-
Licensing Constraints: Breeze TTS’s license restricts commercial deployment, but replacing it with alternative streaming TTS solutions is straightforward.
-
Context Length: Currently limited to approximately one hour (~16k tokens), achievable via modifications such as integrating Pocket TTS on CPU.
-
Turn-Taking Optimization: The turn management system benefits from further tuning to handle casual chat more fluidly.
-
AI Reasoning & Verbosity: The model’s reasoning depth and verbosity can be improved through targeted finetuning, which I plan to incorporate in future iterations.
-
Hardware Compatibility: Though optimized for RTX 4090, support for other high-memory GPUs and AMD options can be developed with community collaboration.
Final Thoughts & Call to Action
Speakrail exemplifies how modern open-source models and clever engineering can bring advanced, private, and low-latency voice assistants within reach of hobbyists and researchers alike. I invite you to explore and experiment with the system:
Visit the GitHub repository to get started.
Your feedback, feature requests, and contributions are highly welcome—together, we can push the boundaries of local AI voice interactions.
Stay tuned for a detailed technical report, and happy building!
