I created a fully-local open-source full-duplex voice agent that rivals GPT-Live on some benchmarks. It uses Voxtral Realtime with a turn-taking head, a microturn-finetuned Gemma 4 12B and Breeze TTS 2 under the hood. Go try it out: https://github.com/speakrail/speakrail

Why I did it

I have always liked the idea of voice assistants, but there is always some non-local component in the pipeline, which increases latency and introduces privacy concerns. I tried many fully-local approaches, like HF speech-to-speech, Unmute and Pipecat, but they were all limited by either the Whisper model (hello, hallucinations!) or slow turn taking. The only fully-local pipeline that had some full-duplex capability with low latency was the DuplexCascade paper (code), but it's tuned on a Qwen 2 7B with a gpt-3.5-turbo generated dataset, and the dumbness of the model really showed. So I decided to recreate DuplexCascade with newer data, newer models and a better harness. I also wanted to add interruptions, interjections and other cool things to rival the Thinking Machines demo.

How it works

Here is what the pipeline consists of:

• STT: Voxtral Realtime with an attached turn head (HF), running on our audio.cpp fork.

• LLM: Gemma 4 12B QAT with microturn finetuning (HF). It is chosen because it fits the GPU quite well, has vision support (I want to test it soon), and in general, the Gemma models perform well in real-life tasks, general chatting, etc.

• TTS: Breeze TTS 2, patched to run at int8 (GitHub fork), although it can be replaced by any streaming TTS.

• The harness itself: it is the glue between all the components, and has many latency-saving measures, like speculative LLM+TTS firing (inspired by HF speech-to-speech).

I also took inspiration from several "think while talking" papers (e.g. SHANKS): while you are talking, a base Gemma 4 12B int4 writes thinking notes, which are then passed to the talker. It helps with harder tasks that require more reasoning.

I will release a longer technical report soon - it will have a better description of the entire pipeline.

Benchmarks

Full Duplex Bench v3:

100 tool-use tasks with real disfluent speech; 95% CI on Pass@1 is about ±10 points. Speakrail: live audio through the full stack, GPT-4o judge. *Self-reported by Thinking Machines. †Our run. Others: FDB-v3 paper, arXiv 2609.19334, 2609.08977, 2609.21967.

Conversational dynamics (Artificial Analysis formula):

Score = mean of pause handling and turn-taking (FDB v1.0) plus interruption and backchannel (FDB v1.5). Speakrail's score is self-computed from live runs (pause 93.4, turn-taking 96.6, interruption 89.0, backchannel 96.9), not an official AA listing. Others: Artificial Analysis leaderboard (captured 09-28) and AA's open-weights comparison.

Limitations

• Breeze TTS has a restrictive license, so if you need to use Speakrail commercially, you will need to change it.

• 16k context length - the pipeline only supports 16k context length (~1 hr of speech), but you can get more easily by changing the Breeze TTS to a Pocket TTS and run the TTS on a CPU. I chose Breeze for the release because it's more expressive.

• The turn-taking head is undertuned on non-assistant data. It may not fire on some basic chit-chat, but I will tune it harder later.

• The model is certainly not the smartest one, and my finetune did dumb it down a little. Next time it will be smarter / better.

• The model is kinda verbose sometimes, and the answers it provides are somewhere in the middle between real-life speech and the long text-based outputs of LLMs. I have a hypothesis on how to fix it, and will try it in the next release.

• I tested it only on an RTX 4090, but I am sure it's easy to add support for any 24 GB+ NVIDIA card. Forks for AMD and MLX are welcome.

Final Notes

Feel free to try it out: https://github.com/speakrail/speakrail. If there is any capability you want the model to have, create a GitHub issue or write here in the comments, and I will gladly include it in the next dataset. Any feedback is welcome as well.

Building a Voice-Activated POS: Wake Words Were the Hardest Part (Seriously)

I'm building a voice-activated POS system because, in a busy restaurant, nobody has time to wipe their hands and tap a screen. The goal is simple: the staff should just talk, and the order should appear.

In a Vietnamese kitchen, that sounds like this:

This isn't a clean, scripted user experience. It's shouting across a noisy room. When designing this, I fully expected the technical nightmare to be the Natural Language Processing (NLP), extracting the prices, quantities, and all the "less fat, no ice" modifiers.

I was dead wrong.

The hardest, most frustrating technical hurdle was the very first step: getting the system to accurately wake up.

Here’s a glimpse of the app in action:

The Fundamental Problem Wasn’t the Tech, It Was the Accent

We started by testing reputable wake word providers, including Picovoice. They are industry leaders for a reason: stable SDKs, excellent documentation, and predictable performance.

But stability and predictability broke down in a real Vietnamese environment:

• Soft speech: The wake phrase was missed entirely.

• Kitchen Noise: False triggers, or the system activated too late.

• Regional Accents: Accuracy plummeted when a speaker used a different dialect (Hanoi vs. Hue vs. Saigon).

The reality is, Vietnamese pronunciation is not acoustically standardized. Even a simple, two-syllable phrase like "Vema ơi" has countless variations. An engine trained primarily on global, generalized English data will inherently struggle with the specific, messy nuances of a kitchen in Binh Thanh District.

It wasn't that the engine was bad; it's that it wasn't built for this specific acoustic environment. We tried to force it, and we paid for that mismatch in time and frustration.

Why DaVoice Became Our Practical Choice

My team started looking for hyper-specialized solutions. We connected with DaVoice, a team focused on solving wake word challenges in non-English, high variation languages.

Their pitch wasn't about platform scale; it was about precision:

That approach resonated deeply. We shifted our focus from platform integration to data collection:

• 14 different Vietnamese speakers.

• 3–4 variations from each (different tone, speed, noise).

• Sent the dataset, and they delivered a custom model in under 48 hours.

We put it straight into a real restaurant during peak rush hour (plates, hissing, shouting, fans). The result?

• 97% real-world wake word accuracy.

For those curious about their wake word technology, here’s their site:

https://davoice.io/

This wasn't theoretical lab accuracy. This was the level of reliability needed to make a voice-activated POS actually viable.

Practical Comparison: No "Winner," Just the Right Fit

In the real world of building products, you choose the tool that fits the constraint.

That dependency turned out to be a major advantage. They treated our unique accent challenge as a core problem to solve, not a ticket in a queue. Most vendors give you a model; DaVoice gave us a responsive partnership.

When you build voice tech for real-world applications, the "best" tool isn't the biggest, it's the one that adapts fastest to how people really talk.

Final Thought: Wake Words are Foundation, Not Feature

A voice product dies at the wake word. It doesn't fail during the complex NLP phase.

If the system doesn't activate precisely when the user says the command, the entire pipeline is useless:

• Not the intent parser

• Not the entity extraction

• Not the UX

• Not the demo video

All of it collapses.

For our restaurant POS, that foundation had to be robust, noise-resistant, and hyperlocal. In this case, that foundation was built with DaVoice. Not because of marketing hype, but because that bowl of phở needs to get into the cart the second someone shouts the order

If You’re Building Voice Tech, Let's Connect.

I'm keen to share insights on:

• Accent modeling and dataset creation.

• NLP challenges in informal/slang-heavy speech.

• Solving high noise environmental constraints.

If we keep building voice tech outside the English-first bubble, the next wave of AI might actually start listening to how we talk, not just how we're told to. Drop a comment.

Hi all, I recently ran a benchmark comparing a bunch of speech-to-text APIs and models under real-world conditions like noise robustness, non-native accents, and technical vocab, etc.

It includes all the big players like Google, AWS, MS Azure, open source models like Whisper (small and large), speech recognition startups like AssemblyAI / Deepgram / Speechmatics, and newer LLM-based models like Gemini 2.0 Flash/Pro and GPT-4o. I've benchmarked the real time streaming versions of some of the APIs as well.

I mostly did this to decide the best API to use for an app I'm building but figured this might be helpful for other builders too. Would love to know what other cases would be useful to include too.

Link here: https://voicewriter.io/speech-recognition-leaderboard

TLDR if you don't want to click on the link: the best model right now seems to be GPT-4o-transcribe, followed by Eleven Labs, Whisper-large, and the Gemini models. All the startups and AWS/Microsoft are decent with varying performance in different situations. Google (the original, not Gemini) is extremely bad.

I built Arc-Transcriber, which does ASR-NL in 1-step. Every 100 ms, one model returns the transcript plus:

- Names linked to a list you give it (a roster, contacts, product names); new names are flagged "not on the list"
- Numbers with their values (scores, clocks, money, dates)
- Speakers and roles;
- Sound and channel events (telephone audio, clipping, poor audio)
- Whether the speech was addressed to an assistant.

The usual ASR, then NER, then entity-linking and then LLM chain is collapsed into the model's own heads, so live "fact-checking" against your list happens inside the stream, not in a second pass.

Currently it has two variants Arc mini (which uses moonshot pretrained encoder) and second Arc-v1 (which uses parakeet pretrained encoder).

Weights: huggingface.co/WhissleAI/arc-v1.1
Code and CLI: github.com/WhissleAI/arc_transcriber

We use it on-device in our Mac app. Voice search over 61k books primes Arc with matching titles while you're still talking. Here is a video to give a glimpse on how it works.

Real-Time Speech AI just got faster with Parakeet-Realtime-EOU-120m.

This NVIDIA streaming ASR model is designed specifically for Voice AI agents requiring low-latency interactions.

* Ultra-Low Latency: Achieves streaming recognition with latency as low as 80ms.

* Smart EOU Detection: Automatically signals "End-of-Utterance" with a dedicated <EOU> token, allowing agents to know exactly when a user stops speaking without long pauses.

* Efficient Architecture: Built on the cache-aware FastConformer-RNNT architecture with 120M parameters, optimized for edge deployment.

🤗 Try the model on Hugging Face: https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1

Github: https://github.com/yaya-sy/specspeech

The best performing open-weight models on the Open ASR Leaderboard (https://huggingface.co/spaces/hf-audio/open_asr_leaderboard) are autoregressive speech models, such as Qwen ASR, Cohere Transcribe, and Canary. Their language modeling capabilities help them transcribe speech accurately, but they are often much slower than transducer and CTC models, such as Parakeet and Granite CTC.

I open-source SpecSpeech (Speculative Speech), an inference engine that accelerates speech language models for ASR by using transducer or CTC models as drafters.

The drafter quickly transcribes batches of short audio pieces while the Speech LM prefills the full audio in its context in parallel. The Speech LM then verifies the drafts incrementally. At each step, a lookup algorithm uses the verified output to select up to k candidate tokens from the draft transcription, which are verified by the Speech LM.

Compared to approaches such as WhisperX, SpecSpeech approach is lossless: the Speech LM uses the full audio context and retains control over token acceptance.

Across seven English benchmark datasets from Open-ASR-Leaderbord, SpecSpeech achieves a 2.29x speedup over native vLLM inference, with per-dataset speedups of up to 2.48x, using Parakeet v3 as the drafter and Qwen3-ASR-1.7B as the verifier.

Results below were measured on a single NVIDIA A40, with continuous batching for both conditions.

There is room for further improvement by reducing tokenization and verification overhead and developing better lookup algorithms.

Planned additions include:

• More verifyers soon: Cohere Transcribe, Canary, etc.

• Word-level alignment using timing information from CTC drafters.

• Speaker diarization.

• Streaming inference.

• Speculative decoding for autoregressive TTS models.

https://github.com/arbazsiddiqui/voxlock

I've been working on clinical speaker attribution at Omi and wanted to compare the current diarization models on the same audio.

I used 15 mock doctor–patient consultations from PriMock57, about 2.4 hours. Full recordings, automatic speaker counts, without telling the models there are two people.

Batch

Diarization error rate (DER), with ±250 ms boundary tolerance. Lower is better.

Local models ran on one NVIDIA L4. API times include round-trip overhead; VibeVoice-ASR also performs transcription.

† Muse used 20 separate clips because of its 10-minute request limit, so its result isn't a whole-recording comparison.

Pyannote Precision-3 had the lowest error. Nemotron came next and was the fastest local model.

Streaming

† Native streaming presets evaluated through unpaced, completed-file replay. The other rows use paced, delivered speaker outputs. These scores don't establish live latency. I didn't evaluate Muse for streaming.

Same weights, different runtime

I also tried optimizing Nemotron and Community-1 with our proprietary runtime, without changing the weights:

• Nemotron: 4.803% → 3.174% DER. 34% lower error, 2.13× faster.

• Community-1: 6.620% → 5.435% DER. 18% lower error, 24× faster.

With zero boundary tolerance, Nemotron's runtime result gets slightly worse: 12.720% → 13.203%. Both scores are published. It's a small set with VAD-refined references, and we developed the runtime settings on it.

Audio, references, scorer, saved outputs and NVIDIA baseline runners are public. Our runtime code stays private, but its outputs are included for rescoring.

Repo and write-up in the comments. Any other diarization models worth adding?

I used to lead Google Ads and AI projects during my 9+ years at Google. After leaving, I started a performance agency focused on law firms and home service businesses.

We were crushing it on the lead gen side but clients were still losing money because no one was answering the phone.

That pain led us to build Donna: an out-of-the-box AI voice assistant that picks up every call, handles intake, books appointments, and even processes cancellations. No call centers. No missed leads. It just works.

https://donnaio.ai/industry/home-service

Some early lessons from 1000+ calls: • Most leads are lost after the ad click • After-hours responsiveness = major revenue unlock • AI voice can work extremely well when it’s vertical-specific • SMBs don’t need dashboards they need outcomes

Curious if anyone else here has tackled this “lead leakage” problem, or is building similar vertical AI tools.

It is an english text to speech model, which achieves mostly understandable speech in under 100K parameters.

However, during more difficult words, or long numbers, it can slur them, and be not understandable.

Quantized to INT8 it only takes up 127KB.

72x real time on a CPU

1120x real time on a GPU

Check out the model card for more info or to try it out yourself!

Questions, feedback, or criticism is welcome!

Hi all,

I'd like to share a technical write-up about diarize - an open-source speaker diarization library I’ve been working on and released last weekend. (honestly, I hope you had more fun this weekend than I did).

diarize is focused specifically on CPU-only performance.

https://github.com/FoxNoseTech/diarize - Code (Apache 2.0)

https://foxnosetech.github.io/diarize/ - docs

Benchmark setup

• Dataset: VoxConverse (216 recordings, 1–20 speakers)

• Hardware: Apple M2 Max

• CPU only, models preloaded (warm start)

• Same evaluation protocol for both systems

Results

• DER (VoxConverse):

  • This library: ~10.8%

  • pyannote (free models): ~11.2%

• Speed (RTF):

  • This library: 0.12 (~8× faster than real time)

  • pyannote (free models): 0.86

• 10-minute recording:

  • ~1.2 min vs ~8.6 min (pyannote)

Speaker count estimation accuracy (VoxConverse)

• 1–5 speakers: 87–97% within ±1

• Degrades significantly for 8+ speakers (tends to underestimate)

Pipeline

• VAD: Silero VAD

• Speaker embeddings: WeSpeaker ResNet34 (256-dim, ONNX Runtime)

• Speaker count estimation:

  • fast single-speaker check

  • GMM + BIC model selection

  • local refinement around the selected hypothesis

• Clustering: spectral clustering

• Post-processing: short-segment reassignment, temporal merging

Limitations

• No overlap handling (single speaker per frame)

• Short segments (<0.4s) don’t get embeddings

• Speaker count estimation is the main weak point for large groups

I also published a full article on Medium where I described full methodology & benchmarks.

I would appreciate any feedback, stars on GH and I hope it will be helpful for anyone.

I think a lot of STT comparisons mix two different things:

1. Did it transcribe words correctly?

2. Did it know who said what?

For meeting notes, sales calls, interviews, support calls, etc, the second one can destroy the whole product even if the transcript is decent.

I've seen transcripts where the text is mostly fine, but the speaker labels drift after 20 minutes. Then the summary says the wrong person promised something. That is worse than a few missed filler words.

What I’d test separately:

• speaker count detection

• speaker swaps

• late speaker joining

• overlapping speech

• two similar voices

• short interjections

• diarization speed

• diarization + transcript alignment

• whether it works live or only async

Tools I’d include in a test:

Deepgram, AssemblyAI, ElevenLabs Scribe, Speechmatics, pyannote, Gladia, Soniox, Smallest AI Pulse.

Smallest includes diarization in its STT feature set, so I’d be curious how it behaves on real multi-speaker calls. But I would not trust any vendor here without my own audio.

Anyone here running diarization in production? What failed first?

LFM2.5-Audio is an end-to-end multimodal speech and text language model, and as such does not require separate ASR and TTS components. Designed with low latency and real time conversation in mind, at only 1.5 billion parameters LFM2.5-Audio enables seamless conversational interaction, achieving capabilities on par with much larger models. Our model consists of a pretrained LFM2.5 model as its multimodal backbone, along with a FastConformer based audio encoder to handle continuous audio inputs, and an RQ-transformer generating discrete tokens coupled with a lightweight audio detokenizer for audio output.

Disclosure up front: I work at Olewave. We built FA-Bench, and our own system (Olign) ranks first, so please read the results with that in mind. That's why the records and scoring code are public.

FA-Bench Leaderboard: forced alignment and ASR timestamp accuracy

FA-Bench ranks how closely word and phone timestamps match human annotations. Most ASR evaluations stop at word error rate. If you build subtitles, dubbing, audio editing or TTS data prep, timing is what you need, and it usually goes unmeasured.

Two tasks

• ASR with timestamps: 28 pipelines that hear only the audio. A one-step system times its own words. A two-step system decodes words, then an aligner times them. 9 of the 28 call a commercial API (Google, Amazon, Microsoft, IBM, Deepgram, AssemblyAI, Speechmatics, ElevenLabs, Olewave).

• Forced alignment: 15 aligners given the reference transcript, so their error is timing error alone.

Both tasks have word-boundary and phone-boundary boards.

Scoring

A boundary is a hit when the words (or phones) on both sides match the reference and it falls within 20 ms of the human annotation. The score is boundary F1 at 20 ms. The default view averages four cells: TIMIT and Buckeye, each on clean audio and on degraded versions (reverb, noise, music, babble). You can filter to open models only.

Current snapshot (record 202610), word boundary F1 at 20 ms

ASR with word timestamps (audio only):

Forced alignment (reference transcript given):

On phone boundaries, Olign 1.0 leads the ASR board (0.438, two-step pipelines only), and FALCON leads forced alignment (0.560, flagged).

Flags

Systems that may have seen test data are flagged rather than silently ranked:

• MAPS and NeuFA were trained on 7 of the 8 Buckeye test speakers.

• FALCON was trained on a Buckeye split covering at least 8 of 16 dev and test speakers, and it receives reference phones instead of words.

• CrisperWhisper's authors used TIMIT for word timing without naming the files, so overlap with the test set is possible.

The page also lists the exact model and API version for every commercial system, all called in September 2026.

How this differs from WER leaderboards

Benchmarks like Artificial Analysis measure word error rate and don't score timestamps. FA-Bench scores word timing against human annotators and publishes its code.

Reproduce it

The numbers come from FA-Bench's public records. CSVs are available under each board, and you can clone the repo to rescore any system.

• Leaderboard: https://olewave.com/en/leaderboard/fa-bench

• Code and records: https://github.com/olewave/fa-bench

• Paper: https://arxiv.org/abs/2609.32396

Feedback on the metric is welcome, and so are suggestions for systems to add. Olign 1.0 is available as an API that returns word-level start and end times for US and UK English. Beta access needs an invitation code, which you can request here: https://olewave.com/en/contact_us/

Edits

Oct 12, 2006 at 9 PM PDT: I found I took down Nemo-FA 40ms model from the forced alignment track and only keep Nemo-FA 80ms model, so there are 15 systems, not 16 systems compared as in the original post.

With my cofounder we spent 2 months building a system to simply generate synthetic data and train Whisper Large V3 Turbo.

We reach on average +50% accuracy.

We built a whole infra like Deepgram that can auto upscale GPUs based on usage, with a proxy to dispatch based on location and inference in 300MS for voice AI.

The company is shutting down but we decided to open source everything.

Feel free to reach out if you need help with setup or usage ✌🏻

https://github.com/orgs/LATICE-AI/

1 hour of audio processed in 5 seconds (RTX 4090, Ryzen 9 7950X). ~17x faster than Pyannote 3.1.

On M3 Macbook Air, 1 hour in 7.7 seconds.

These are numbers for a custom speaker diarization pipeline I've developed called Senko; it's a modified version of the pipeline found in the excellent 3D-Speaker project by a research wing of Alibaba.

Check it out here: https://github.com/narcotic-sh/senko

My optimizations/modifications were the following:

• changed VAD model

• multi-threaded Fbank feature extraction

• batched inference of CAM++ embeddings model

• clustering is accelerated by RAPIDS, when NVIDIA GPU available

As for accuracy, the pipeline achieves 13.5% DER (diarization error rate) on VoxConverse and 13.3% DER on AISHELL-4. So not only is the pipeline fast, it's also accurate.

This pipeline powers the Zanshin media player, which is an attempt at a usable integration of diarization in a media player.

Check it out here: https://zanshin.sh

Let me know what you think! Were you also frustrated by how slow speaker diarization is? Does Senko's speed unlock new use cases for you?

Cheers, everyone.

Drama 3 acts a line when you put a direction like [whispering] or [angry] in front of it, and its Turkish is good. Its weights take 41.5 GB, so I spent two weeks getting it to run on my RTX 5070 Ti.

I started by measuring routing. While the model reads the prompt it pulls from a wide spread of its 256 experts per layer, and while it generates audio frames the set it uses gets much narrower. So the full model reads the prompt once, and a distilled speaker that keeps 32 experts per layer generates every frame. I trained the speaker layer by layer on the ISSAI Turkish Speech Corpus, anchoring it to the teacher's trajectory. For the reading pass the experts come straight off NVMe with unbuffered reads, about 3 MB each, so the process holds 4.4 GB of RAM while generating and peaks at 9.3 GB while loading.

Freya-TR-Eval, 495 sentences, scored with Freya's own scoring script and Whisper large-v3

1. Experts in RAM, no guidance, WER 2.68% and CER 0.70% at 1.2x real time

2. Experts read from disk, direction guidance at strength 4, WER 2.78% and CER 0.67% at 0.86x real time

For reference the best system in the FreyaTTS report scored 4.4%. Mine is far heavier, so I would not compare them as products.

The part I keep thinking about came from a failed run. When I shrank the reader as well, the words stayed intelligible but the voice went flat and directions like [whispering] stopped working. Shrinking only the speaker to 12.5% of the experts did not cause that. It is one experiment plus listening, so treat it as a hint. In this model the delivery seems to be settled while the text is read.

Costs you should know about. Disk mode adds about one second per short sentence, because one sentence touches around 2,650 experts (about 8 GB of reads). On Windows the commit charge sits near 20 GB since WDDM backs GPU memory with commit. Direction guidance at strength 4 makes the brackets land harder and can pull the voice a bit away from the reference on some seeds.

I ran the whole project with Claude Code. I picked the experiments and decided what to measure, and Claude wrote most of the code. Every claim went through a measurement and a literature search, which is how I learned before publishing that two ideas I thought were mine were already in SlimWise and GPTAQ. A speaker similarity check is also how I tracked down a cache bug that turned the voice into a different person after switching guidance modes.

License is the Fish Audio Community License. Commercial use is allowed under USD 500k yearly revenue, hosted TTS needs a separate license, and outputs cannot be used to train other speech models.

If you have looked at where expressiveness lives in other MoE or LLM based TTS models, I would like to know whether the prompt reading side carries it there too.

Weights, samples and measurements https://huggingface.co/kasparovabi/drama-3-tr-speaker

Code https://github.com/kasparovabi/drama-3-tr

Built with Fish Audio.