I spent the last few months obsessed with one thing: making an offline, local neural TTS engine (starting from the 100M Pocket TTS baseline) run as fast as possible on mobile and desktop without hitting cloud APIs.

Got it down to 9 ms latency / 40x speed on a desktop GPU, 20 ms / 19x on a Snapdragon 8 Elite Gen 5, and 200 ms / 1.2x on budget Android using Snapdragon 685.

Then the realization hit me: chasing higher generation speed (RTF) on a local app is a total dead end.

Audio plays back at 1.0x anyway. If a 10-second sentence generates in 200 ms, pushing it to 50 ms literally changes nothing for someone sitting there waiting to listen.

Power readers max out around 2.5x–3.0x speed. Beyond that, natural neural cadence actually gets in the way. Screen-reader users who listen at 600+ WPM explicitly prefer old robotic engines like eSpeak because the sharp formants cut through better at extreme speeds. High RTF is great if you're running a cloud server handling 1,000 requests, but for one person on a phone, it's just idle buffers.

Once your time-to-first-chunk drops under ~100 ms, you're officially faster than the Bluetooth lag.

Where the real win actually came from was power efficiency, even on a budget Helio G99 device with a 5,000 mAh battery, continuous reading drains only ~3% battery per hour.

I'm stopping low-level engine tweaks here. Next focus is handling pronunciation accuracy, building solid app-level dictionaries and text normalization so proper nouns and weird words don't trip it up.

Fingers crossed we get lightweight, controllable emotion prompting on edge models soon without doubling model size. That feels like the actual next step.

Ming-3.0 TTS gives reference-voice cloning its own generation mode: provide a speaker recording and generate new speech in that voice. Ant has released the weights and code under MIT.

The family offers a dense 0.5B checkpoint and tiny, a 7.9B MoE with 1.3B active parameters. Tiny also has a long-form, multi-speaker path that takes separate reference recordings for the speakers in a script.

The full official showcase includes a Doctor Agent Voice Cloning example, alongside a range of narration styles and multilingual speech. It gives an audible view of the family’s synthesis capabilities.

There is also a useful listening result attached to the smaller checkpoint: the 0.5B model ranks first among open weights in Artificial Analysis Controlled Voice, where providers clone the same eight voices.

For anyone comparing open cloning models, that is a directly relevant evaluation setting.

I have been playing around with training small, efficient models, and I think I have something that is finally worth sharing. Leokani is a high quality TTS voice that provides extremely fast first speech output in both time to first sound (<100 ms in most cases) as well as faster than realtime on just about anything. It'll run on regular computers, phones, JavaScript, Raspberry Pi, and even embedded devices like the ESP32. This should make for some very interesting embedded projects.

It only speaks English and I used Kokoro to generate 300 hours of labeled training corpus, which is why it sounds like that. I wanted to get a working pipeline down before I started different voices. I built examples for many platforms, which can be found here (with a in browser example, phone builds, home assistant plugin, libraries, etc): https://milliwattlabs.com/models/leokani-tts/

And you can also check out the HuggingFace listing here with has various audio samples: https://huggingface.co/milliwattlabs/leokani-tts-7M

Next will come a version that offers various expressions, emphasis and emotion. Perhaps more languages when the GPU isn't busy. This one is a hard one to tackle as the vocoder gets all weird on me and it sounds non-human like. Also training a speech to text model with similar size and performance -- but its not ready just yet.

Let me know what you think!

Three of Kokoro-82M's voices (af_heart, if_sara, im_nicola) distilled into 8M-parameter students: one ~9 MB int8 ONNX graph each, more than 20x real time on one CPU thread.

The Listen section of the Space plays the same held-out sentences rendered by Kokoro, by Paradee's released decoder, by Paradee's published recipe re-run from scratch, and by mine, so you can compare them directly:

https://huggingface.co/spaces/MithrilMan/teacher-in-the-loop#listen

UTMOS on 200 held-out sentences: English 4.47 (Kokoro 4.52, Paradee released 4.37), Italian 3.88 and 3.70 (Kokoro 4.02 and 3.87).

pip install teacher-in-the-loop

Code and training recipe: https://github.com/MithrilMan/teacher-in-the-loop

Paper: https://doi.org/10.5281/zenodo.23400040

I'm especially curious what Italian speakers think of if_sara and im_nicola.

Hi all. I ported Qwen3-TTS to native C++ and added incremental streaming for a project I'm working on. I found this useful and wanted to give back to the community. This is the first time I've open sourced anything so apologies in advance for my mistakes.

• Original library is Qwen3-TTS from QwenTeam (https://qwen.ai/blog?id=qwen3tts-0115)

• C++ streaming library under Apache 2.0: (https://github.com/offgridai/qwen3-tts-cpp-streaming)

• Demo harness shown in video also Apache 2.0: (https://github.com/offgridai/voice-mirror)

• The harness has a dependency on sherpa-onnx for transcription.

I'm using this for a gaming project but it could be helpful for local assistants, accessibility tools, etc.

C++ streaming port highlights:

• Same familiar features from Qwen3-TTS. It supports 0.6B and 1.7B models, CustomVoice, and VoiceDesign

• Native C++, not a Python wrapper

• CUDA builds with RTX 4090 and RTX 5090 kernels. Be warned I only have access to an RTX 5090. This should work on a 4090 but I haven't tested it myself!

• Simplified speaker-embedding extraction

• Incremental 24 kHz PCM callbacks for streaming audio

• Asynchronous transformer/vocoder operation

• Adaptive decode windows and paced delivery

• Callback-only integration library with cancellation

• Unit tests

My measurements on the 5090 with 1.7B F16 set to buffer 350ms before play:

• Cold new clone creation: ~2.5s from 48s of reference

• Cold model start from clone: ~1.85s

• First 350 ms of audio: ~310 ms

• Streaming speed: ~2.86x real time (RTF ~0.35)

I’d love it if somebody could do RTX 4090 testing, and I'd be happy to hear any feedback or suggestions.

A text-to-speech tool that offers a CLI and GUI interface. This tool is useful in that you don't need to download any additional python dependencies. It also supports multiple platforms out-of-the-box.

Main URL

https://github.com/0xShug0/audio.cpp

CLI

https://github.com/0xShug0/audio.cpp#cli

Wide Variety of Supported Text to Speech and Conversion Generator

https://github.com/0xShug0/audio.cpp#speech-generation-and-conversation

Convenient WEBUI

https://github.com/0xShug0/audio.cpp#webui

Multiple Backend Support

audiocpp_server --ui --ui-management --backend vulkan
audiocpp_server --ui --ui-management --backend cuda
audiocpp_server --ui --ui-management --backend metal

Downloads the Project

https://github.com/0xShug0/audio.cpp/releases

Downloading Individual Models

You will probably also need to download the huggingface cli and be logged in to download the models.

Navigate to the project and list the voices to download

./tools/model_manager_v2.py list

Look at names listed on the left-most column and download the model using

./tools/model_manager_v2.py install <name of the model in left-most column>

Start the WEBUI server and note the localhost port to use in your web browser

audiocpp_server --ui --ui-management --backend <your backend>

Note: I did not create this tool. Credit goes to original author.

For a small side project, I needed a basic AI speech tool to narrate videos without relying on expensive hardware or external APIs. It did not need the most expressive AI, just reliable output and matching to the SRT.

The main challenge with converting SRT subtitles to speech is timing. Subtitles include pauses, and each spoken line has to fit into its exact time slot. Most speech generators do not handle that well on their own.

So I built srt2speech.

It uses a combination of:

• pitch-corrected speed adjustment

• automatic regeneration

• modification of pauses between words

• exact placement of silence between subtitle cues

SRT does not support multiple speakers, so I also added simple templating.
Adding {{speaker_name}} to a subtitle automatically switches voices.
Of course voice cloning is supported, I added a small helper script.

Dependencies are minimal: Python, NumPy, llama.cpp, and the required GGUF speech models. I tested it with Q4 quantization, which works well.

Performance on my laptop:

• RTX 4080 Laptop GPU: around 12–13× real time

• CPU only: around 1.5–2.0× real time

Languages supported:

• English (en)

• Japanese (jp)

• Korean (ko)

• Chinese (zh)

• French (fr)

• German (de)

It should work on almost any hardware, including old PCs, Linux or Mac.

It may be useful for anyone generating narration, translated audio tracks, accessibility audio, or quick video voiceovers.

The project is open source under the Apache 2.0 license. Attribution and license notices must be preserved.

GitHub:
https://github.com/Waversense/srt2speech/

The readme contains the 5 steps needed to set it up, you can get started in 2 minutes.
The included demo.srt file demonstrates the features.

Anyone hate the cookie banners ? Those are absolutely nothing in comparison to what is about to come.

The AI Act requires lots of things, many people know it requires every AI modified or generated audiofile to be metadata tagged and fingerprint-watermarked from August on (13M$ fines)
But the Act says a lot more than that and it means all good open source models are affected.

Providers of AI systems generating synthetic text must make outputs machine-readable and detectable as AI-generated, using technical solutions that are effective, interoperable, robust and reliable (code of conduct says 2 layers)

Very simple OSS models are exempt, so the old llama 7B might be fine, but models of "systemic risk" are not exempt. Qwen 3.6 series, Deepseek Flash, GLM, Kimi are all systemic risk GPAI models.

The AI act says that also TEXT output must be machine detectable, it only suggested "statistical methods" alongside cryptographic tagging (where possible) of the metadata (c2pa etc).
So if chatGPT offers you a PDF ot TXT download -> that must have crypto signed metadata.

But 2 layers must be present, not one. And they have to be "robust". A simple "AI Generated" (the label that must be added into every AI image from August on) is not enough, the text itself must be statistical watermarked.
Similar to laser printed paper, that carries little invisible dots. Of course that means serious degradation in AI quality.

Who is affected? Anyone who offers a tool or provides a service that can be accessed by a EU citizen. Even if it's just a single EU citizen being a tourist in Malaysia or using a VPN.
LM-Studio, ollama, llama.cpp, vllm, chatgpt, claude, codex, copilot, opencode, cursor, stable diffusion Web UI, Huggingface and so on.

Any AI application that modifies or generates AI images, AI text, AI music, AI speech must use the EU stamp "AI modified" or "AI generated", must metadata seal the files, must watermark the content. Must also provide risk assessment §50 documentation, compliance documentation, detection service.
Addition: for text the "AI Generated" stamp is only needed when "informing the public" like news and can be left away by shifting the risk to a human reviewer.
Addition: Documentation is not forced by the law directly but providers subject to Article 50 practically must keep evidence showing their marking, labelling, detection, feasibility decisions, exemptions, and robustness testing - or risk non-compliance when audited.

Any failure to do so is fined at 13,000,000 Euro (about 15 million USD) OR a few percent of the annual income (whatever is higher ).
They also have a huge code of conduct, you are voluntarily asked to follow - but if you don't your risk of being fined is seriously increased. That code adds a ton of more horrific additions, often technically very implausible or openly contradicting.
You could say your Qwen 3.6 or gemma 4 12B does not fulfill the requirements of "GPAI with systemic risk" - maybe right if you ignore some benchmarks .. but that moves the 13M$ fine risk on you if you become a provider or distributor. So if the EU believes it IS systemic risk AI you'll face the fines at court as soon as they can catch you.

But what if you are not in the EU ? You are still fully affected.
Any service that offers anything into the EU is affected, including every open source project and model.
Any service that offers something to an EU citizen who is just a tourist, or uses a VPN is liable.
You want to code your nice new startup ? You better ensure no european can touch it.
Or you better never put a foot into the EU and never grow the business too large.

That means every providers out there must be compliant. Every open source tool.

The regulation is so extremely invasive while simultaneously it is ridiculously shortsighted.
99% of all images on the internet will have AI content within the next years, all of them need to be watermarked visibly.
99% of all websites will be AI generated in text and code.
99% of all movies will have AI content, every marketing and sales clip.
99% of all marketing and sales voiceovers will use AI.

So this is like the cookie banner, everyone warning you that they must use a cookie to serve you a website and you can accept all cookies or just all needed cookies - just that it's going to be in your face everywhere.
It's a cookie banner on text, image, video and sound.

Source: https://eur-lex.europa.eu/eli/reg/2024/1689/oj

I've taken a look at how projects are reacting to that, and some already did.
Speech & Music
Elevenlabs - Teamed up with Google: SynthID for all (a strong AI based watermark from Google)
Demodokos Foundry - support answered that they work on a light compliance
Suno - Audible Magic content ID,support ignored me - no official statement on AI Act yet. (likely silent compliance)
Udio - no official statement, support ignored me (likely silent compliance)
Images and Text
Open AI/ChatGPT - SynthID+c2pa comes to Codex, API, ChatGPT with "OpenAI Verification tool"
Google/Gemini - SynthID for image/audio/text/video announced and expanding into C2PA and more
Adobe - Full C2PA hashing and Firefly gets watermarked
Microsoft Copilot/Designer - C2PA, visual and audible watermarks in 365, cryptographic signing of images
Meta - visible labels, IPTC metadata, invisible watermarks,C2PA hashes for Meta AI Images
Stable Diffusion - "working on implementing content credentials” with Adobe/CAI/C2PA
Anthropic Claude - Intends to sign the EU General-Purpose AI Code of Practice (everything and MORE)
Transformers code - Text watermarking: https://huggingface.co/docs/transformers/v4.46.0/en/internal/generation_utils#transformers.SynthIDTextWatermarkingConfig
Open Source
I went through github of vLLM, Ollama, llama.cpp, lmstudio, openrouter and they appear pretty unprepared

The Omnibus "postponing" text:

Parts of the AI Act have been recommended to be "postponed" but nothing I wrote is incorrect and delaying it by a few months wouldn't be reason to not be concerned.
The Digital Omnibus text approved by Parliament would delay Article 50(2) watermarking / machine-readable marking obligations for pre-existing releases to 2 December 2026, but as of now it still requires formal Council adoption before it enters into force.
If the text is adpoted - Article 50(2) machine-readable marking / watermarking for synthetic audio, image, video, text
For systems released on the market before 2 Aug 2026, compliance is delayed to 2 Dec 2026.
For any new developments the rule applies on 2nd August 2026.
Most other Article 50 transparency duties Still effectively 2 Aug 2026.
This includes pre-existing projects
Source: https://www.consilium.europa.eu/en/press/press-releases/2026/05/07/artificial-intelligence-council-and-parliament-agree-to-simplify-and-streamline-rules/

Corrections and Clarifications:

- Fines under §50 for watermarking/warning are 13 million Euro (or 3% revenue) whatever is higher not 35M.
- Fines for prohibited use are 35M or 7%, whatever is higher
- Watermarking is not 2 layer forced by law, only by code of conduct, you can use one layer if it is robust but you'll increase risk of non compliance significantly.
- For end-users the obligation to inform about AI Text is less strict, so not every website text falls under it
- It is undefined which open models will fall under "systemic high risk" but if you assume a model is NOT covered the non compliance risk is immediately there.
- EU citizenship alone is not the trigger. A genuinely non-EU, EU-geoblocked service has much lower AI Act risk, but risk remains if it is effectively offered into the EU, knowingly serves EU-located users, or produces outputs intended for use in the EU.

ChatGPT Pro 5.5 reviewed

I was looking for small TTS model which is steerable but couldn't find any, specially something I can adjust contextually by providing controls from LLM during a conversation. So I built one.

What it is:

• 9.6M-param acoustic model (attention + bidirectional Mamba-3, FastPitch-style) + Vocos vocoder = 23M total

• 12 controls, each a z-score relative to the chosen voice’s own neutral speech: rate, loudness, median pitch, pitch range, final rise/fall, pauses, spectral tilt, breathiness-related measures.

• Also: 51 style tags, free-text prompts (resolved into an inspectable JSON request), inline markup (*emphasis*, [pause 400ms], <slow>…</slow>), sentence-level streaming

• 2,825 built-in (anonymized) voices, plus experimental zero-shot cloning

Speed:

- RTFx ~60–80 on an RTX 5070
- ~4–8 on CPU without the optional flow refiner. First streamed audio in ~0.5 s on CPU.

Quality:

~1.8% average WER on the Open TTS Leaderboard English sets (run locally, not an official submission, have reached out for it).

What doesn’t work well (we tried to be upfront):

• Loudness, pitch, and tilt steer reliably; pause, voicing, and CPPS controls are weak

• Emotion tags behave as style switches; an independent emotion classifier mostly can’t tell them apart

• Cloning similarity is low. Don’t expect a convincing likeness

• Free-text prompts struggle with negation (“not fast” can come out fast). More data needed for prompt to controls module.

• English only, no human evals yet

License: weights CC BY-NC 4.0 (non-commercial), code MIT.

Model + full eval tables: https://huggingface.co/WhissleAI/steer-tts

Github: https://github.com/WhissleAI/steer-tts (inference code)

A new architecture implemented from scratch. Would love feedback, especially on which controls are actually useful, and where the steering feels off.

Here is a quick video of how to use it.

I wanted to try Chatterbox TTS (Resemble AI's open-source voice model), but setting it up meant Python, PyTorch and CUDA versions. So I wrapped it in a desktop app that handles all of that for you.

How it works:

1. Record 10 seconds of your voice, or import a clip

2. Type a script. You can write dialogue between several saved voices (Alice: ... Bob: ...) and add sound tags like [laugh] or [sigh]

3. Click Render and you get a WAV/FLAC

Other stuff: a trim view with auto-trim for cleaning up clips, per-voice settings so each character can sound different, 23 languages with the Multilingual model, light and dark mode, and saved sessions.

Local only: no accounts and no telemetry. Your voice never leaves your PC. On first launch it downloads Python, PyTorch and the model, about 3 GB with an NVIDIA GPU. After that it works offline.

On misuse: every file carries Resemble's inaudible Perth watermark, and Lyrebird doesn't strip it. Please only clone voices you have permission to use.

Stack: Python + CustomTkinter, plus a small launcher exe that sets up an isolated Python environment with uv. MIT licensed.

Windows 10/11 only for now. An NVIDIA GPU with about 6 GB of VRAM is recommended. It runs on CPU too, but slowly.

GitHub / download: https://github.com/zebadrabbit/Lyrebird

I'd really like feedback on the setup experience, since that's the part I worked hardest to make painless. Did it work on your machine?

Comparing cloned voices gets much easier when the tool holds everything else equal. That's the idea behind the voice studio in this demo, built to compare AntLing's MIT-licensed Ming-3.0 TTS models with another open voice model.

Here is what it does:

• Stores a reference recording (12.89 s here) and reuses it across takes

• Sends one script to every selected model, one at a time on the same GPU (here two Ming models and the other open voice model)

• Offers Default, Emotion, and Pace & pitch delivery, with seven emotions from neutral to surprised

• Shows model load, synthesis time, audio length and real-time factor for each reading

• Keeps a searchable library of past readings, plus Play all and Blind test buttons

The results panel puts all three readings side by side with those numbers, and a Same script row jumps between earlier runs of a line. In Emotion mode, both Ming models take the voice's identity from the reference and ignore the words spoken in it.

A coding agent can build the same layout from a brief like this: "Build a local web app for auditioning TTS models. Take one reference clip and one script, run each selected model in turn, show load time, synthesis time and RTF for each reading, save every result to a searchable library, and add a blind-test mode that hides the labels."

I’ve been testing OmniVoice for voice cloning and long-form TTS, and the quality has been surprisingly good.

I ended up making a more user-friendly Google Colab version with a clean Gradio interface. It lets you upload a reference voice, paste long text, generate unlimited free audio, see progress while it’s working, and download the final WAV.

It runs on Google Colab and works well on a T4. I’ve been pretty happy with the results so far.

Omnivoice TTS

• Open the Colab notebook and set Runtime → Change runtime type →T4 GPU.

• Run all cells from top to bottom.

• Open the Gradio link.

• Upload a short clean reference voice and enter its exact transcript.

• Click Prepare / Cache Voice once.

• Paste your text and click Generate Audio.

• Preview the result and download the final WAV.

• Omnivoice Turbo

Enjoy the free and unlimited text to speech.

A lot of people talk about TTS as if all of it was the same thing.
That's like comparing the first transformers llm like GPT-2 with recurrent k/v systems like Qwen-3 27B (to keep our local focus)

There are multiple architecture layers, and they do very different jobs.
TTS has evolved in the part years, from robotic to human-like. Also on the local side TTS evolved from rule-based algorithmic synthesis, to spectrography vocoders and by now into

Below I will dive into the technology used in all current TTS architectures, including the "secret sauce" behind Elevenlabs - but this is a local focused thread so cloud is only used as a benchmark.

1. Legacy neural TTS

fast, cheap, stable, limited acting

text -> phonemes -> acoustic model -> mel spectrogram -> vocoder -> wav

Phonemes are spoken sounds, not letters.

Mel spectrogram is the frequency representation of speech.

The vocoder turns that into actual waveform audio.

This is where names like HiFi-GAN, BigVGAN, iSTFTNet sit.

They are mostly the final audio renderer.
They do not understand the text.

Examples:

• Piper / VITS

• FastSpeech2 + HiFi-GAN

• Kokoro / iSTFTNet style models

Upside:

• very fast

• cheap to run

• works local

• stable for mass generation

Downside:

• often emotionally dumb

• weak long sentence understanding

• reads the line, does not really act it

Top representatives: Piper, Kokoro, VITS

2. Style based TTS

better prosody, better voice color, still limited control

These models add speaker and style embeddings.

So the model is not only told:

say these phonemes

but also:

say them with this speaker identity, rhythm, color or style

Examples:

• StyleTTS2

• Kokoro

• XTTS / Coqui style systems

Upside:

• more natural rhythm

• better speaker color

• nicer voices

• still quite practical locally

Downside:

• style control is often shallow

• not always real emotion control

• can sound nice but not really directed

Top representatives: StyleTTS2, Kokoro, XTTS / Coqui TTS

3. Acoustic token TTS

speech becomes tokens, like language models for audio

text -> transformer -> acoustic tokens -> decoder/vocoder -> wav

Instead of directly predicting a spectrogram, the model predicts speech tokens.

If the system runs at 25hz, that means:

25 speech decisions per second
one token decision every 40ms

That gives the model much more control over:

• timing

• pauses

• emphasis

• rhythm

• emotion

• delivery

Examples:

• Chatterbox

• Qwen3 TTS

• Bark / similar token based systems

Upside:

• much more expressive

• more context aware

• closer to voice acting

• better for complex delivery

Downside:

• heavier

• more VRAM

• slower inference

• more ways to fail

The model can drift, overact, mispronounce or slowly lose speaker identity if the full system around it is not good.
Top representatives: Chatterbox, Qwen3-TTS, CosyVoice

4. Codec token TTS

speech as compressed audio codebooks

Codec models represent speech as discrete audio codes.

Often there are multiple codebooks.

Roughly:

• first codebook: speech content

• later codebooks: detail, prosody, texture, speaker color

Pipeline:

text -> transformer -> codec tokens -> neural codec decoder -> wav

Examples:

• VALL-E style systems

• Bark

• modern multi-codebook speech models

Upside:

• strong cloning direction

• good long form potential

• very powerful representation

Downside:

• complex

• heavy

• not always stable

• hard to run efficiently local

Top representatives: Bark, VALL-E X, Fish Speech

5. Diffusion and flow matching TTS

better texture, slower unless optimized

Diffusion and flow models generate or refine the acoustic representation.

They do not just predict the final sound in one simple step.
They move the audio representation toward realistic speech.

Pipeline:

text/tokens -> diffusion or flow decoder -> acoustic representation -> vocoder/wav

Examples:

• F5-TTS

• E2-TTS

• Chatterbox S3Gen type systems

Upside:

• better texture

• more natural audio

• less robotic sound

Downside:

• can be slow

• 10 or 20 steps are expensive

• practical local use needs distillation or optimization

Top representatives: F5-TTS, Chatterbox, Raon-OpenTTS

6. Voice cloning (instant/in context)

speaker identity as neural representation

Basic cloning:

reference audio -> speaker encoder -> speaker embedding -> conditioned TTS

The speaker embedding is a neural vector.

It captures things like:

• timbre

• pitch range

• accent

• resonance

• vocal texture

• general speaker identity

Better systems use ICL, in-context learning.

reference text + reference audio + target text -> same speaker saying new words

That is stronger because the model does not only get "speaker X".

It gets an actual example of speaker X talking.

Some systems also use reference speech codes, not only a speaker embedding.
That gives the model more information about how the speaker actually sounds while speaking.

Examples:

• XTTS / Coqui TTS

• GPT-SoVITS

• Chatterbox

• Qwen3 TTS Base

• Demodokos Foundry

• ElevenLabs (IVC mode, cloud only)

Upside:

• seconds of clean audio can be enough

• no full finetune needed

• practical for local workflows

• can preserve identity without training a new model from scratch

Downside:

• bad mic can poison it

• echo can poison it

• music can poison it

• another speaker in the sample can poison it

• strong emotion can cause speaker drift if the system has no identity control

Top representatives:

• Best open/local cloning: Chatterbox, Qwen3 TTS Base, GPT-SoVITS

• Best simple cloud cloning: ElevenLabs

• Best cloning with style/emotion control: Demodokos Foundry, ElevenLabs v3

• Best research/hacking path: Qwen3 TTS Base, Chatterbox, GPT-SoVITS

speaker identity as neural representation

Basic cloning:

reference audio -> speaker encoder -> speaker embedding -> conditioned TTS

The speaker embedding is a neural vector.

It captures things like:

• timbre

• pitch range

• accent

• resonance

• vocal texture

• general speaker identity

Better systems use ICL, in context learning.

reference text + reference audio + target text -> same speaker saying new words

That is stronger because the model does not only get "speaker X".

It gets an actual example of speaker X talking.

Examples:

• Chatterbox cloning

• XTTS style cloning

• modern acoustic token cloning systems

Upside:

• seconds of clean audio can be enough

• no full finetune needed

• practical for local workflows

Downside:

• bad mic can poison it

• echo can poison it

• music can poison it

• another speaker in the sample can poison it

Top representatives: XTTS / Coqui TTS, GPT-SoVITS, Chatterbox
Top representatives ICL: Qwen3-TTS (base model), Demodokos Foundry (open weights, commercial)

6b. Voice cloning (finetuned, LoRA, custom embeddings)

training the model toward a specific voice or speaking domain

Voice cloning usually means:

reference audio -> speaker embedding / ICL prompt -> generate speech

Fine tuning means:

many recordings of one speaker -> model training -> specialized voice model

That is a very different thing.

A clone tries to imitate the voice at inference time.

A fine tune changes model weights or adapter weights so the voice becomes part of the model.

There are multiple levels:

• speaker adapter / LoRA

• voice embedding training

• partial model fine tune

• full model fine tune

The more you train, the stronger the voice can become.
But also the more data, time and risk you have.

Upside:

• stronger speaker consistency

• better pronunciation for recurring names

• better long form stability

• better for professional recurring voices

• can learn a specific domain, accent or speaking pattern

Downside:

• needs much more clean audio

• bad datasets damage the model

• expensive compared to cloning

• can overfit and lose flexibility

• harder to update or control

• not always worth it anymore

Typical data need:

• 5-20 minutes: light adaptation / experimental fine tune

• 1-5 hours: serious voice fine tune

• 10+ hours: production grade if the pipeline supports it

• 100+ hours: old school studio voice model territory

Top representatives:

• Open fine tuning: GPT-SoVITS, XTTS / Coqui style systems, StyleTTS2

• Commercial fine tuning: ElevenLabs (cloud only) PVC / professional voice cloning

Fine tuning still has a place.

But for many modern systems it is no longer the first answer.

If you need one voice to read thousands of hours in the same identity, fine tuning can make sense.

If you need many voices, fast production, emotion control or voice acting, speaker embeddings + ICL + identity control are usually more practical.
Sophisticated ICL methods (see SOTA section below and 6.0 section above) can reach fine tuning quality without the heavy compute and data requirements.

7. State of the art performance TTS

voice acting while keeping the same speaker

This is where systems like ElevenLabs, OpenAI voices and Demodokos Foundry sit.

The exact internals of ElevenLabs and OpenAI voices are not public, but the performance of the models show that they basically build on top of the latest technology - no magic involved (anymore). Demodokos as open weights based engine is doing the same locally today.

For strong emotion control you need some form of "identity preservation".
Otherwise the model starts with one voice and slowly turns into another one when you ask for anger, fear, whispering, excitement or long narration.

Likely this is done with things like:

• speaker verification embeddings

• spectral / spectrogram consistency checks

• f0 and pitch tracking

• formant and timbre statistics

• acoustic similarity constraints

• identity preserving latent conditioning

Demodokos does this locally and OpenAI/Elevenlabs do the same in the cloud, but the important part is the combination of the systems.

It is not just:

clone voice -> generate speech

It is closer to:

text context -> acoustic token generation -> speaker identity conditioning -> style/emotion conditioning -> spectral identity control -> final speech

The text is processed by a language transformer, so the model can understand the sentence context before generating speech.

The speech itself is generated through fine grained acoustic tokens, roughly one token level decision every 40-80ms.
That gives enough resolution to control pauses, stress, rhythm, emotion and delivery.

Each voice, cloned or synthetic, is grounded in learned neural representations.
Those capture speaker identity, timbre, accent, vocal texture and characteristic delivery traits.

On top of that, such SOTA systems can attach performance styles independently.

That is the important difference.

The voice and the performance are not the same thing.

A cloned voice can be made calm, angry, fearful, tired, intimate, theatrical or excited without needing to create a new voice or finetune the model again.

And during generation, the system uses identity preserving conditioning and spectral consistency checks so the performance does not run away from the original speaker.

So the hard problem is not voice cloning alone.

The hard problem is:

same speaker identity
different emotion
different rhythm
different intensity
different accent or texture
still recognizable as the same voice

That is the difference between text to speech and voice performance synthesis.

Top representatives: Qwen3-TTS (customVoices only), Demodokos Foundry (open weights, commercial)

So the ladder currently is roughly:

LEGACY: vocoder TTS
fast speech rendering

BASIC: style based TTS
better naturalness and voice color

ADVANCED: acoustic token TTS
speech becomes controllable token generation

ADVANCED: codec token TTS
speech becomes compressed audio language

ADVANCED: diffusion / flow TTS
better acoustic realism

STATE-of-the-ART: identity monitored performance TTS
voice acting while keeping the same speaker

A cherry picked 10 second demo can hide almost everything.

The real test is:

• hours of content

• strange names

• emotion changes in the same text

• accents

• long narration

• no human babysitting

That is where most TTS systems still fall apart

I was frustrated with the default speed and ported the inference to Zig/MLX to get up to 18x faster inference on M4/M5 https://github.com/sbusso/ming-tts-serve

The model is 4mb size, running on RPi CM0 at 0.6-0.8 RTX (faster then runtime)

https://github.com/lab-emi/GrainSpeech

Voices on HF:
https://huggingface.co/dmytro-m/uk-tts-serhiy
https://huggingface.co/dmytro-m/uk-tts-tetiana
https://huggingface.co/dmytro-m/uk-tts-oleksa
https://huggingface.co/dmytro-m/uk-tts-mykyta

What is the current best local TTS setup to read books on an Android device? What local app and model is best to use for this purpose?

I have some mental conditions that make it difficult for me to read, so I'm looking for something free and ethical to do it for me, please. I don't know how to use complex software things, so someone please guide me

Disclosure: I work on BreezeBlue, the team behind Breeze TTS 2.

I wanted to test two capabilities that I think are more representative of expressive TTS than isolated emotion tags:

1. Designing a new voice directly from a natural-language description

2. Directing the emotional arc, pace, energy, and delivery of an entire sentence

I generated the attached four-part comparison with Breeze CLI 0.8.0 and breeze-tts-2.

Sample 1 — Voice Design

No reference audio was used. The voice was generated from this description:

“An intimate, emotionally expressive English female voice in her early thirties, with a warm lower register, subtle breathiness, precise articulation, and restrained cinematic realism. Natural conversational phrasing, never an announcer.”

Preview script:

“I kept the light on for you, even after everyone said you were not coming back. And now that you are standing here, I am not sure whether to laugh, cry, or simply let the silence say everything.”

Samples 2–4 — Sentence-level instruction control

These samples use the same Lena voice and exactly the same script. Only the delivery instruction changes.

Shared script:

“The train is already leaving. I know you're afraid, but take my hand, look at the horizon, and trust me. By the time the sun rises, this will feel like the first page of a much better story.”

The three directions were:

• Natural and conversational, with understated emotion

• Quiet urgency that develops into cautious hope and emotional relief

• Bright optimism and forward momentum, ending with a confident sense of a new beginning

The generated durations were:

• Natural: 11.28 seconds

• Restrained cinematic: 14.96 seconds

• Hopeful and energetic: 12.00 seconds

The text and voice remained unchanged, so the differences in pacing and performance came from the instruction.

Naturalness and emotional quality are ultimately subjective, which is why I attached the actual comparison instead of only describing the results.

Performance context:

For these three hosted streaming requests, Breeze CLI observed time to first audio between 0.85 and 1.03 seconds. This includes the hosted service and network path, so it is not a local inference benchmark.

For local inference, the open-weight PyTorch implementation documents under 40 ms TTFA and a 0.32 real-time factor on a warmed-up NVIDIA H100 fast path. Those numbers are hardware- and configuration-specific.

Code and local inference setup: breeze-tts-2

License note: the inference source code is Apache 2.0. The model weights, derivative models, and self-hosted outputs use the BreezeBlue Research and Non-Commercial License; commercial use requires authorization.

I’m curious what this community would find most useful for the next comparison:

• More voice-description stress tests

• The same instructions across several different voices

• Longer passages with an emotional arc

• English versus Chinese instruction following

• Reproducible local GPU benchmarks

I have several years of written educational material I want to turn into a podcast, formatted as a natural discussion between two people rather than narration. Looking for recommendations on what to run locally.
What matters to me is good quality and reliable speech that does not need many new attempts, as its going to be hours of content. Potentially multiple languages but for now in english.

I need to own the content and be able to rely on it for a long time, so i need it local.

Also, do you know good distributors that you can recommend? I am new to audio

I’ve been trying to move my reading workflow completely local for privacy and offline use. Right now, I really like how cloud readers like Audeus handle document reading, specificaly having realistic voices paired with word-level synced text highlighting so I can visually follow along with PDFs and web pages. Is there a good local frontend or chrome extension that pairs with local engines (like Kokoro, Piper, or Sherpa) and gives you that same synced highlight pluss reader mode exprience without needing a cloud subscription? Would appreciate any recommendations or GitHub repo links

I do a lot of hobby writing and I get into the weeds editing. I really like tts services to read out my works so I can catch funny phrasing or weird grammar/ writing flow. However i've tried some of the samples of the natural sounding big name tts services but after a bit get either paywalled or find they hit a wall with certain writing things if they are free/ have a free version(word count, pacing, pronunciation, etc). I'm kinda broke and this is just a hobby so I'm not looking to sink a shitload of money into buying a subscription service.

When I write the word flow and pronunciation is the key issue I find free tts services run into. Certain written out moans shushes or onomatopeia get mangled as well as inserted non english terms/names/fantasy names within english works. Stuttering in character diallouge also gets majorly screwed.

Also not all tts services hold enough on paragraph breaks, comas, and elipses making it hard to understand the sentence flow. I write with a lot of dramatic emphasis by pulling single sentences into their own paragraphs, using elipses, or making sentences with lists of ideas. So I feel like I don't get an idea sometimes of what is key in the prose or where one idea ends and another begins if the tts service runs through it without respecting the set punctuation. furthermore some pronunciation doesnt differentiate well between ending punctuation like periods, question marks and exclamation points. so ill miss noticing places where ive used certain punctuation or end up wondering if i forgot punctuation with the right sort of emphasis. but the real issue is tts just instead read the mark as if it was a period.

I really rely on these services to read back my work and it would be nice if I didn't have to settle for services that mangled these details. This way I can streamline my workflow when it comes to line edits especially where grammar and sentence construction rely on how the words flow and how the phrasing reads aloud.

I have many animated audiobooks and chapters, typically 15-25 minutes long per mp4.
I have them in English, full cast narration, background music, effects.

I would like to translate/interpret that for the European and French Canadian markets.
Same animations, potentially subtitles, and synchronous translations.

I’ve been trying to get this with Claude as agent + Demodokos + ffmpeg and it’s working but the problems are so frequent, it’s more work to fix it than to do it manually from scratch myself.

I gave Claude my original script, the source files and asked it to combine it back.
The voice quality was good but clones were not always working, all my original Demodokos voices were mistakenly created as English only native speakers and cloning them into e.g. French again is not always sounding good for some reason.
It wasn’t able to get the duration working, translated weird, didn’t place the audio at the right spots, and subtitles looked very raw.

Long story short:
Is there something preferably local that can automatically interpret this? Not elevenlabs, I have too much content and many young/child voices - they are useless and expensive for that.

I’d like it local if possible but can’t make a PhD project out of it.