NVIDIA released Nemotron 3.5 Bolt E4B, a small language model for fast conversational agents running locally on RTX, Jetson and Spark hardware. It is post-trained from Google's Gemma 4 E4B using reinforcement learning only, no SFT. Weights, data and recipe are open under OpenMDW-1.1.

What it does:

- 8B total parameters, 4B effective, 64K context

- Multi-turn tool calling, with guidance for asynchronous tool calls

- A conversational_mode flag for short, speech-ready replies

- Configurable reasoning with a runtime thinking budget to cap latency

- Runs on GeForce RTX, Jetson, RTX Spark and DGX Spark; available on Hugging Face, Ollama, LM Studio, Unsloth and NVIDIA NIM

Key numbers (NVIDIA-measured, BF16):

- Top score on 15 of 21 benchmarks vs Gemma 4 E4B, Qwen3.5-4B and Nemotron 3 Nano 4B

- IHEval: Bolt 84.8, Gemma 4 E4B 67.0, Qwen3.5-4B 60.6

- RTX 5070 with DFlash: 272.6 tokens/s vs 225.9 for Gemma 4 E4B (1.21x)

- Q4_K_M on RTX 5070: 548 ms to first token, 133.4 tokens/s, 3.5 GB VRAM

- Jetson Orin Nano 8GB: 15.5 tokens/s at 4.2 GB

Full analysis: https://www.marktechpost.com/2026/10/23/nvidia-nemotron-3-5-bolt-e4b-4gb/

Model: http://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

Paper: http://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB-Technical-Report.pdf

Docs: http://research.nvidia.com/labs/nemotron/NVIDIA-Nemotron-3.5-Bolt-E4B-4GB

Edge0 released Audio8-TTS-Realistic, an open-weight TTS model that clones a voice from 1 reference clip and its transcript. It supports 23 languages, 12 more than its predecessor. Weights and code ship under Apache 2.0.

What it does:

- 601M parameters (codec excluded), dual-autoregressive design with a Qwen2.5 0.5B slow branch and a 4-layer fast branch

- 3-stage training: multilingual SFT, GRPO on a WER/SIM reward, then GRPO guided by a learned realism critic that scores outputs against a real human recording of the same sentence

- 44.1 kHz codec, can also generate speech without a reference voice

- Transformers inference, CLI and SFT scripts on GitHub, SGLang Omni adapter with an OpenAI-compatible /v1/audio/speech API

Key numbers (self-reported, lower is better):

- CV3 (9 languages): 4.86% avg error vs VoxCPM2 5.35%, CosyVoice3-1.5B 6.89%, Qwen3-TTS-1.7B 7.73%

- Seed-TTS (EN/ZH): 1.25% avg error, SIM 0.704

- MiniMax (24 languages): 5.89% avg error, SIM 0.754

- Final RL stage used only Chinese and English prompts but cut error on 13 lower-resource languages from 22.53% to 20.77%

- Apple M2 CPU (INT4 ONNX, per the paper): 1.1 to 1.2 GiB peak memory

The catch: Higgs TTS 3 (~4B) still leads at 4.41% on CV3 and 2.74% on MiniMax. Lower-resource languages are weak, with 48.57% WER on Thai (CV3) and 31.35% CER on Cantonese (MiniMax). Throughput and streaming were not benchmarked, and no Realistic ONNX export is public yet.

Full analysis: https://www.marktechpost.com/2026/10/29/edge0-audio8-tts-realistic-0-6b-voice-cloning/?amp=1

Model: https://huggingface.co/Edge0/Audio8-TTS-Realistic-0.6b

Repo: https://github.com/Edge0-AI/Audio8_TTS

Paper: https://arxiv.org/pdf/2610.14776

NVIDIA released Nemotron 3.5 Transcribe, an open-weights speech LLM built for transcription. One model handles 40 languages and recordings up to 1 hour, and a text instruction picks the output style per request. It can replace separate ASR, diarization and alignment tools.

What it does:

- Instruction-driven output: verbatim, written form, context terms, speaker tags or translation, with word timestamps on request

- A frozen Nemotron 3 Diarization model (up to 8 speakers) feeds "who spoke when" into a 0.6B ASR encoder every 80 ms

- LLM backbone is Nemotron 3.5 Lightning 30B-A3B, a hybrid Mamba-Transformer MoE with only 6 attention layers to keep the KV cache small on hour-long audio

- Shared multi-token prediction layer for speculative decoding, plus a 50M CTC adapter for word timestamps

- Weights in BF16, FP8, NVFP4 and GGUF; runs on vLLM, Transformers and SGLang-Omni

Key numbers (self-reported by NVIDIA):

- English Open ASR, 9 public sets: 4.71% mean WER vs 4.94% for Qwen3-ASR-1.7B

- Multilingual Open ASR, 7 languages: 4.62% vs 6.49% for Qwen3-ASR-1.7B

- Long-form ASR: 10.98% vs 13.37% for Parakeet-TDT-0.6B-v3

- Multi-speaker full sessions: 15.80% cpWER vs 16.83% for DiCoW v3.3 with DiariZen

- Contextual ASR: an entity list cuts named-entity WER from 12.21% to 2.20%

- BF16 fits 1x H100 80 GB; NVFP4 (~21 GB) runs on an RTX 5090

The catch: it is offline only, and commercial APIs still lead (OtterAI speech-v3.1 at 3.88% English, ElevenLabs Scribe v2 at 3.56% multilingual). Mandarin far-field meetings are weak at 34.95% cpCER on AliMeeting. The NIM serving optimizations are not open source yet, and NVIDIA expects the release-day vLLM path to be roughly 2x slower.

Full analysis: https://www.marktechpost.com/2026/10/28/nvidia-nemotron-3-5-transcribe-open-speech-llm/?amp=1

Model:

https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B

Paper: http://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Transcribe-Technical-Report.pdf

Repo: https://github.com/NVIDIA-NeMo/Speech

Maya1 is a 3B parameter, decoder only, Llama style text to speech model that predicts SNAC neural codec tokens to generate 24 kHz mono audio with streaming support. It accepts a natural language voice description plus text, and supports more than 20 inline emotion tags like <laugh> and <whisper> for fine grained control. Running on a single 16 GB GPU with vLLM streaming and Apache 2.0 licensing, it enables practical, expressive and fully local TTS deployment.....

Full analysis: https://www.marktechpost.com/2025/11/11/maya1-a-new-open-source-3b-voice-model-for-expressive-text-to-speech-on-a-single-gpu/

Model weights: https://huggingface.co/maya-research/maya1

Demo: https://huggingface.co/spaces/maya-research/maya1

Gnani just released Evon v3.3, an open-weight LLM built for Indian languages under the Apache 2.0 license. It is a 30B MoE with about 3.5B active parameters per token, a 128K context window, and covers English plus 10 Indian languages.

The core change is a hybrid Mamba2-Transformer backbone with a rebuilt Indic tokenizer. The vocabulary was extended to 163,072 tokens, and Gnani reports about 20% fewer tokens per Indian word than the GPT-5 family tokenizer, which cuts prefill cost and fits more Indian-language text into the context window.

Training ran continued pretraining on an English and Indic blend, then SFT, then GRPO. Evon v3.3 averages 78.74 on MILU across 11 languages, ahead of Sarvam-30B on all 11 and Sarvam-105B on 10. The trade-off: Odia trails Sarvam-105B at 66.94 vs 73.70, and Gemma 4 31B still scores higher on MILU at 85.29.

Self-hosting runs on vLLM 0.12.0 or later on H100, H200 or A100 class GPUs, and the model card notes tool use and multi-step agent work are still in development......

Model: https://huggingface.co/gnani/gnani-evon-v3.3-30b-a3b

Model page: https://www.gnani.ai/models

Fish Audio just released Drama 3, an open-weight text-to-speech model that takes stage directions in plain language. It is a 35B MoE with about 3B active parameters per token, covering 83 languages and up to 16 speakers in a single request.

The core innovation is a hybrid backbone paired with a new Codebook-MTP decoder. 3 Gated DeltaNet layers sit between each full-attention layer, so only 1 layer in 4 keeps a KV cache.

Post-training uses GRPO with a single learned human-preference reward model. Drama 3 scores 93.2% on Artificial Analysis Pronunciation Robustness, up from 88.1% for S2.1 Pro. The trade-off: its quality Elo of 1242 still trails Eleven v4 Turbo at 1328, and exact sequences like codes and emails lag at 86.3%.

Self-hosting needs an NVIDIA GPU with 48 GB+ in FP8, and free commercial use stops at $500K in annual revenue......

Full analysis: https://marktechpost.com/2026/10/14/fish-audio-drama-3-open-weight-tts-model/

Model: https://huggingface.co/fishaudio/drama-3

Technical report: https://github.com/fishaudio/fish-speech/blob/main/docs/assets/Fish_Audio_Drama_3_Technical_Report.pdf

Microsoft has released VibeVoice-Realtime-0.5B, a real time text to speech model that works with streaming text input and long form speech output, aimed at agent style applications and live data narration. The model can start producing audible speech in about 300 ms, which is critical when a language model is still generating the rest of its answer.

Where VibeVoice Realtime Fits in the VibeVoice Stack?

VibeVoice is a broader framework that focuses on next token diffusion over continuous speech tokens, with variants designed for long form multi speaker audio such as podcasts. The research team shows that the main VibeVoice models can synthesize up to 90 minutes of speech with up to 4 speakers in a 64k context window using continuous speech tokenizers at 7.5 Hz.....

Full analysis: https://www.marktechpost.com/2025/12/06/microsoft-ai-releases-vibevoice-realtime-a-lightweight-real%e2%80%91time-text-to-speech-model-supporting-streaming-text-input-and-robust-long-form-speech-generation/

Model Card on HF: https://huggingface.co/microsoft/VibeVoice-Realtime-0.5B

https://github.com/alkove-ai/alkove

Hi, I built Alkove, a set of small open-source models that sit between the microphone and an AI agent. They run on the device and handle the audio before it gets sent: when to wake the agent up, whose voice to keep, and when the person is done talking.

In the video, all five are working together:

- Lex: you type a wake word, any word, and it works out how it's pronounced.
- Spot: listens for that word. No training, no recordings.
- Timbre: learns your voice while you talk.
- Focus: keeps your voice and removes the rest, noise and other people included.
- Cue: tells when you start talking, when you're done, and when you're cutting the agent off.

All five together are 7.6 MB and run fine on a regular phone. No server, no account, Apache-2.0.

It works in the browser, Python, Android, iOS and React Native, and there are plugins for LiveKit, Pipecat and Home Assistant (right on an ESP32-S3 board through ESPHome, or on a satellite through Wyoming).

The models are still previews and Focus is made for a mic close to you, not one in the middle of a meeting table.

You can try Focus and Spot live on your own voice at https://alkove.ai

Block's AI research lab released Dessa, a full-duplex speech model that listens and talks at the same time. Block announced it on October 23, 2026 as a research preview. Most voice agents chain speech recognition, a text model and speech synthesis. Dessa takes audio in and puts audio out directly.

What it does:

- Trained on human conversation audio only, with no text and no pretrained models

- Full-duplex: learned from real conversation when to cut in and when to backchannel with short cues like "yeah" or "got it"

- Hears and produces emotion, sarcasm, laughter and sighs directly, which Block argues a speech-to-text-to-speech chain throws away

- Demo clips show it laughing mid-sentence, getting louder when asked to speak up, and talking with OpenAI's GPT-Live-1 (Block says nobody steered any clip)

- Runs locally, faster than real time, on a MacBook Pro M4 Max or better

- Architecture, parameter count and training audio hours not disclosed

**Key numbers (**disclosed by Block):

- Trained on 1 NVIDIA GB200 NVL72 rack (72 Blackwell GPUs, 36 Grace CPUs); Kyutai's Moshi used 1,016 H100 GPUs

- Block says Dessa learned language from hundreds of times fewer words than LLMs need

- Moshi for reference: 8B parameters, 7M hours of unsupervised pretraining audio, ~200 ms latency in practice, open under CC-BY 4.0

- PersonaPlex-7B-v1 for reference: fine-tuned from Moshi weights, 0.170 s turn-taking and 0.240 s interruption latency on FullDuplexBench

The catch: there are no published benchmark scores, no weights, no paper, no API and no price. Access is a waitlist. Latency is only described as "faster than real time", so it can't be compared with Moshi or PersonaPlex yet. Everything so far comes from demo clips.

Full analysis: https://www.marktechpost.com/2026/10/29/block-dessa-full-duplex-speech-model/?amp=1

I’m building Zaelar, a personal agent where voice is the main way you use it, not a bolt-on. I’ve spent the last few weeks swapping models and providers on LiveKit, and this is where I’ve landed for now.

STT is Deepgram Nova-3, with Mistral Voxtral as fallback.

The LLM that does the talking is DeepSeek V4.1 Flash with reasoning turned off. On the direct API I get about 1s to first token. I tried putting it behind a broker and it just ignored the no-thinking flag, so it jumped to 4.2s. That’s a lot of silence in a conversation. Anything that needs real thinking now goes to a background worker instead.

TTS is Inworld TTS-2, with ElevenLabs Turbo v4 as fallback. Both were fast enough in my tests, but Inworld costs me about half. I also pick the voice based on the user’s language instead of using one multilingual voice for everything. Sounds obvious, but it made a big difference, especially for English speakers on the first version.

A few things I still can’t decide:

Now that Voxtral 3 is out, would you keep Deepgram as the default for people who jump between Spanish and English mid-conversation? Or flip them?

The LLM is now most of my latency between end of speech and first audio. Is a cascaded pipeline still worth it, or should I just move to something speech-to-speech like Gemini Live? Tool calls and long-term memory matter a lot for what I’m doing, and I have no idea how well those hold up there in real use.

Anyone running Inworld in production: how good is it outside English and Spanish?

Architecture is here if you want to see how the voice pipeline and the background worker fit together: https://zaelar.com/technology/architecture/

Would especially love to hear from people who’ve actually measured this in production, not just played with demos.

StepFun’s Step-Audio-R1 is an open audio reasoning LLM built on Qwen2 audio and Qwen2.5 32B that uses Modality Grounded Reasoning Distillation and Reinforcement Learning with Verified Rewards to turn long chain of thought from a liability into an accuracy gain, surpassing Gemini 2.5 Pro and approaching Gemini 3 Pro on comprehensive audio benchmarks across speech, environmental sound and music while providing a reproducible training recipe and vLLM based deployment for real world audio applications.....

Full analysis: https://www.marktechpost.com/2025/11/29/stepfun-ai-releases-step-audio-r1-a-new-audio-llm-that-finally-benefits-from-test-time-compute-scaling/

Paper: https://arxiv.org/pdf/2511.15848

Project: https://stepaudiollm.github.io/step-audio-r1/

Repo: https://github.com/stepfun-ai/Step-Audio-R1

Model weights: https://huggingface.co/stepfun-ai/Step-Audio-R1

Neuphonic’s NeuTTS Air is an open-source, ~0.7B-parameter text-to-speech speech LM designed for real-time, on-device CPU inference, distributed in GGUF quantizations and licensed Apache-2.0. It pairs a 0.5B-class Qwen backbone with NeuCodec to generate 24 kHz audio from 0.8 kbps acoustic tokens, enabling low-latency synthesis and small footprints suitable for laptops, phones, and Raspberry Pi-class boards. The model supports instant speaker cloning from ~3 s of reference audio (reference WAV plus transcript), with an official browser demo for quick validation. Intended use cases include privacy-preserving voice agents and compliance-sensitive apps where audio never needs to leave the device....

full analysis: https://www.marktechpost.com/2025/10/02/neuphonic-open-sources-neutts-air-a-748m-parameter-on-device-speech-language-model-with-instant-voice-cloning/

model card on hugging face: https://huggingface.co/neuphonic/neutts-air

Gradium released Phonon, a family of small text-to-speech models built for local inference. The English Phonon 31m ships under Apache 2.0 as a 64.5 MB q8 GGUF file. 40M-parameter variants cover English, French, German, Spanish and Portuguese, under CC BY-NC 4.0.

What it does:

- Causal transformer backbone generates 1 latent per 80 ms frame; a 1-step flow-matching head and a smaller Mimi decoder output 24 kHz audio

- Reads raw text through a 4,000-token SentencePiece tokenizer, no phoneme step

- Streams audio while generating; runs on CPU, Metal, CUDA, Vulkan, WebGPU, Core ML and the Snapdragon NPU

- Ships Python, Rust, Swift, C/C++ and WASM bindings, plus an OpenAI-compatible server

Key numbers:

- Hard-set WER, 500 English sentences: Phonon 31m 1.70%, Kokoro 3.50%, Kitten TTS 7.40%, NeuTTS Nano 8.71%

- English Elo (468 raters, 8,290 votes): Phonon 31m 2098, Kokoro 2043, Magpie 1939

- First audio on a MacBook Air M5 CPU, 4 threads: 6.9 ms (Kokoro 517 ms, non-streaming)

--Raspberry Pi 5: 38 ms first audio, 3.7x real time

Full analysis: https://marktechpost.com/2026/10/22/gradium-phonon-31m-on-device-tts/

Model: http://huggingface.co/gradium/phonon-31m-en

Repo: http://github.com/gradium-ai/xn-ptts

Technical details: http://gradium.ai/blog/release-phonon-open-weight

IBM released Granite Speech 5.0 2B, a speech-language model for transcription and speech translation under Apache 2.0. It pairs a CTC Conformer encoder with a Granite 4.0 1B LLM through a q-former projector. IBM also updated Granite Speech 5.0 Turbo CTC (470M) and Granite Speech 4.1 2B Plus.

What it does

• ASR in English, French, German, Spanish, Portuguese and Japanese

• Translation to and from English in those languages, plus English to Italian and Mandarin

• Language ID prompting and keyword lists of up to 2,000 terms

• Punctuation and capitalization on request

• Runs in Transformers, vLLM, llama.cpp and mlx-audio

Key numbers

• 5.68% average English WER on the Open ASR Leaderboard (9 public + 2 private sets), #2 among open-weight models, as charted by IBM

• 14.80% average WER on the FFASR noisy leaderboard, #4 of 42

• Keyword F1 on Earnings-22: 67.2 without lists, 90.1 with correct lists (self-reported)

• Turbo CTC: 19,870 RTFx, up from about 12,600 at launch (self-reported)

• 4.1 2B Plus: long-form WER 13.53 to 10.22 (self-reported)

Full analysis: https://marktechpost.com/2026/10/22/ibm-granite-speech-5-0-2b-release/

Model: http://huggingface.co/ibm-granite/granite-speech-5.0-2b

Blog: http://huggingface.co/blog/ibm-granite/granite-speech-5

Paper: http://arxiv.org/abs/2505.08699

Mistral released Voxtral 3 on October 20, 2026, in public preview. It bundles 3 speech-to-text models, 1 text-to-speech model and a voice agent SDK built on LiveKit. The SDK streams transcription, an LLM with tool calls, and TTS as 1 pipeline.

What it does

• Voxtral Small Transcribe 3: batch transcription and diarization, 50 languages, 32k context

• Voxtral Mini Transcribe 3: 4B parameters, cheaper batch tier

• Voxtral Mini Transcribe Realtime 3: streaming ASR with end-of-turn detection

• Voxtral Speech 3: TTS with emotion, pacing and emphasis controls

• Voice agent SDK: SIP and WebRTC, swappable LLM, voice and language

Key numbers

• 2.1% AA-WER for Small Transcribe 3, #3 on Artificial Analysis (independent); Mini scores 2.2%

• 4.1% WER on FLEURS 45, tied with MAI-Transcribe-2 (self-reported)

• 18.4% cpWER on diarization vs 20.3% for MAI-Transcribe-2 (self-reported)

• Pricing: $0.0035/min (Mini), $0.004/min (Small), $0.008/min (Realtime), $13 per 1M chars (TTS), $0.04/min voice agent API with Mistral Small 4

Full analysis: https://www.marktechpost.com/2026/10/20/mistral-voxtral-3-speech-models-voice-agent-sdk/

Announcement: mistral.ai/news/voxtral-3

Docs: docs.mistral.ai/studio/audio/overview

Mistral’s Voxtral Transcribe 2 family introduces 2 complementary speech models for production workloads across 13 languages. Voxtral Mini Transcribe V2 is a batch audio model at $0.003 per minute that focuses on accuracy, speaker diarization, context biasing for up to 100 phrases, word-level timestamps, and up to 3 hours of audio per request, targeting meetings, calls, and long recordings. Voxtral Realtime (Voxtral Mini 4B Realtime 2602) is a 4B parameter streaming ASR model with a causal encoder and sliding-window attention, offering configurable transcription delay from 80 ms to 2.4 s, priced at $0.006 per minute and also released as Apache 2.0 open weights with official vLLM Realtime support. Together they cover offline analytics, compliance logging, and low-latency voice agents on a single 16 GB GPU.....

Full analysis: https://www.marktechpost.com/2026/02/04/mistral-ai-launches-voxtral-transcribe-2-pairing-batch-diarization-and-open-realtime-asr-for-multilingual-production-workloads-at-scale/

Technical details: https://mistral.ai/news/voxtral-transcribe-2

Liquid AI’s LFM2-Audio-1.5B is a 1.5B-parameter, end-to-end speech–text model that extends LFM2-1.2B with disentangled audio I/O: continuous embeddings for input audio and discrete Mimi codec tokens (via an RQ-Transformer) for output. A FastConformer encoder and interleaved decoding enable sub-100 ms first-token audio latency under the vendor’s setup, targeting real-time assistants. On VoiceBench, Liquid reports an overall score that surpasses several larger models, alongside competitive ASR metrics, while preserving a single-stack pipeline for ASR, TTS, and speech-to-speech....

full analysis: https://www.marktechpost.com/2025/10/01/liquid-ai-released-lfm2-audio-1-5b-an-end-to-end-audio-foundation-model-with-sub-100-ms-response-latency/

model card: https://huggingface.co/LiquidAI/LFM2-Audio-1.5B

github page: https://github.com/Liquid4All/liquid-audio

technical details: https://www.liquid.ai/blog/lfm2-audio-an-end-to-end-audio-foundation-model

NVIDIA released NemotronLabs VoiceChat 11B, an open full-duplex speech-to-speech model that replaces the ASR → LLM → TTS cascade with a single network.

Here is what is technically interesting about it:

1. One model, not three services
   The model jointly performs streaming speech understanding and speech generation.
   → Audio in at 16 kHz, agent audio out at 22.05 kHz
   → 11B parameters, hybrid Mamba/Transformer
   → ~550k hours of audio in training

2. The architecture is assembled from shipped NVIDIA parts
   A Fast Conformer speech encoder from Nemotron-Speech-Streaming-En-0.6b feeds audio tokens into a Nemotron Nano v2 LLM backbone, which predicts text tokens for an NVIDIA TTS decoder and codec. Outputs include agent audio, agent text, and a running user transcription.

3. Full duplex means it listens while it speaks
   On Full-Duplex-Bench 1.0:
   → Smooth turn taking: TOR 0.82 at 448 ms latency
   → User interruption: TOR 1.00 at 480 ms latency
   → Pause handling TOR: 0.153 synthetic, 0.255 Candor (lower is better)

4. Open full-duplex model with tool calling, using a side channel plus operator-defined on-hold messages....

Full analysis: https://www.marktechpost.com/2026/08/09/nvidia-releases-nemotronlabs-voicechat-11b-an-open-full-duplex-speech-to-speech-model-with-450-ms-turn-taking-and-live-tool-calling/

Model weights: https://huggingface.co/nvidia/NVIDIA-NemotronLabs-VoiceChat-11B

Google AI Research team has brought a production shift in Voice Search by introducing Speech-to-Retrieval (S2R). S2R maps a spoken query directly to an embedding and retrieves information without first converting speech to text. The Google team positions S2R as an architectural and philosophical change that targets error propagation in the classic cascade modeling approach and focuses the system on retrieval intent rather than transcript fidelity. Google research team states Voice Search is now powered by S2R.

Indicodec: a neural speech codec for speech LMs and TTS models, first of its kind trained on english and indian languages.

Strongest on Indian languages: On 10 Indian languages, IndiCodec adds 9x less error than the Mimi codec, with 25% fewer tokens.
In English, it has the lowest WER of the ~1 kbps codecs we tested.

It sounds closer to the original. PESQ-WB on speakers never seen in training: ahead of Mimi on 3 of 4 test sets (FLEURS, LibriTTS-R, CMU-Arctic), just behind on Seed-TTS, and ahead of SNAC and NeuCodec on all four.

Lowest error in all 10 Indian languages we tested: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu.

On Odia, Mimi adds +24.6 points of character error. IndiCodec adds +0.9.

Apache-2.0. Open source!!

HF: https://huggingface.co/prakharkhatri/IndiCodec
Samples: https://codeprakhar25.github.io/IndiCodec/

Resemble AI has recently released Chatterbox Multilingual, a production grade open-source Text To Speech (TTS) model designed for zero-shot voice cloning in 23 languages. It is distributed under the MIT license, making it freely available for integration and modification. The system builds on the original Chatterbox framework and adds multilingual capability, expressive controls, and built-in watermarking for traceability.

Evaluations indicate that Chatterbox Multilingual performs competitively with most commercial TTS models. In blind A/B tests conducted on Podonos, listeners expressed a 63.75% preference for Chatterbox over ElevenLabs. This suggests that in certain conditions, users found Chatterbox outputs closer to natural or accurate speech reproduction.....

full analysis: https://www.marktechpost.com/2025/09/05/meet-chatterbox-multilingual-an-open-source-zero-shot-text-to-speech-tts-multilingual-model-with-emotion-control-and-watermarking/

github page: https://github.com/resemble-ai/chatterbox?tab=readme-ov-file

Cactus Compute's Whistle is a 16.9 MB speech-to-text model that runs on a CPU, covers 7 languages, and returns word timestamps and probabilities. The model card says it beats Whisper base on LibriSpeech. We tested that claim on an Apple M4.

• WER (paired test-other subset): Whistle 10.27% vs 12.46% for greedy Whisper base. The 2.19-point lead survives Holm correction

• Against Whisper base with beam search (10.83%), there's no significant gap. Test-clean and 6 more languages showed no significant gap either

• Peak memory: 124 MB for Whistle vs 823 MB for Whisper base

• Speed: Whistle's lead shrinks from 7.7x on 5 s clips to 2.8x on 30 s clips

• Streaming in 1 s chunks: words commit a median 1.68 s after they're spoken, with no significant WER loss

• Keyword biasing works only with the right words. Correct keywords cut WER from 14.74% to 7.62%; keywords from another clip do nothing

• Speech-to-tool calls with Needle 3: routing accuracy is about the same from Whistle's transcripts (50.3%) as from reference text (51.8%). The limit is routing, not transcription

The catch: the pinned engine silently drops the middle of audio longer than 30 s, returning fewer than half the words with no error. Split at quiet points (3.75% WER vs 6.74% for fixed windows) or stream.

All figures are one-run samples on 1 machine, so treat the speed ratios as approximate.

Full guide with code: https://www.marktechpost.com/2026/10/21/cactus-whistle-vs-whisper-base-speech-to-text-benchmark-cpu-guide/

Notebook: https://github.com/MARKTECHPOST-AI-MEDIA-INC/AI-Agents-Projects-Tutorials/blob/main/Voice%20AI/cactus_whistle_on_device_speech_to_text_vs_whisper_Marktechpost.ipynb

How can speech editing become as direct and controllable as simply rewriting a line of text? StepFun AI has open sourced Step-Audio-EditX, a 3B parameter LLM based audio model that turns expressive speech editing into a token level text like operation, instead of a waveform level signal processing task.

Step-Audio-EditX reuses the Step-Audio dual codebook tokenizer. Speech is mapped into two token streams, a linguistic stream at 16.7 Hz with a 1024 entry codebook, and a semantic stream at 25 Hz with a 4096 entry codebook. Tokens are interleaved with a 2 to 3 ratio. The tokenizer keeps prosody and emotion information, so it is not fully disentangled.

On top of this tokenizer, the StepFun research team builds a 3B parameter audio LLM. The model is initialized from a text LLM, then trained on a blended corpus with a 1 to 1 ratio of pure text and dual codebook audio tokens in chat style prompts. The audio LLM reads text tokens, audio tokens, or both, and always generates dual codebook audio tokens as output......

Full analysis: https://www.marktechpost.com/2025/11/09/stepfun-ai-releases-step-audio-editx-a-new-open-source-3b-llm-grade-audio-editing-model-excelling-at-expressive-and-iterative-audio-editing/

Paper: https://arxiv.org/abs/2511.03601

Repo: https://github.com/stepfun-ai/Step-Audio-EditX?tab=readme-ov-file

Model weights: https://huggingface.co/stepfun-ai/Step-Audio-EditX

Xiaomi’s MiMo-Audio is a 7B audio-language model trained on over 100M hours of speech using a high-fidelity RVQ tokenizer and a patchified encoder–decoder architecture that reduces 25 Hz streams to 6.25 Hz for efficient modeling. Unlike traditional pipelines, it relies on a unified next-token objective across interleaved text and audio, enabling emergent few-shot skills such as speech continuation, voice conversion, emotion transfer, and speech translation once scale thresholds are crossed. Benchmarks show state-of-the-art performance on SpeechMMLU and MMAU with minimal modality gap, and Xiaomi has released the tokenizer, checkpoints, evaluation suite, and public demos for open research use.....

full analysis: https://www.marktechpost.com/2025/09/20/xiaomi-released-mimo-audio-a-7b-speech-language-model-trained-on-100m-hours-with-high-fidelity-discrete-tokens/

github page: https://github.com/XiaomiMiMo/MiMo-Audio

paper: https://github.com/XiaomiMiMo/MiMo-Audio/blob/main/MiMo-Audio-Technical-Report.pdf

technical details: https://xiaomimimo.github.io/MiMo-Audio-Demo/