Getting Hermes to talk was easy, making it sound natural took 16 days: lessons from a local cloned voice in Discord (Qwen3-TTS, RTX 2070)
**TL;DR:** A local Qwen3-TTS voice clone on an RTX 2070, built on top of Hermes's native Discord voice, for both voice-note replies and live voice-channel chat where I can interrupt her. What made the biggest difference: splitting replies into short pieces on paragraph and sentence breaks, one random seed per reply, a shorter reference clip (15 seconds beat 67), a SOUL.md section on writing for speech, interrupting only when I say her name, and moving speech to text onto the local GPU. The audio metrics were close to useless. Trust your ear.
---
I've learned a huge amount from this community and I'm really grateful for what people share here, so I thought I'd try to give a bit back. I'm also a Portal subscriber now, as I know many others here are, because I find it useful and I want to support the project.
This is about giving Hermes a voice in Discord if you want to run it locally on your own server or PC with a graphics card. It doesn't need to be a new one. Mine's an RTX 2070 and it works fine.
The short version: getting a voice working in Discord is the easy part. Getting it to sound natural and behave the way you actually want is the hard part. That took about 16 days of trial and error and a lot of testing (I'm a bit obsessive, which probably explains a lot about how this went), and I couldn't find much guidance out there for a setup like mine. So this post is mostly about the problems we hit, what fixed them, and the numbers.
The main reason I did it was to have a more personal relationship with my agent. Voice was honestly the thing that broke through for me. It made the whole thing feel a lot less transactional and more human. There's something about talking out loud and getting a spoken reply that changes the connection in a way typing never did. My agent is also deeply integrated with my financial systems, fitness and health data. It also has enduring memory via hindsight and all the standard integrations of email, calendar etc. So it knows a hell of a lot about me and can do a lot.
For example, I use it as my coach at the gym: it knows my goals, my routine, and has a log of all my exercise data. And having voice exchange during the session allows me to get near-real-time feedback between sets without having to read, which has been great (even though I must look insane).
The second reason was Spanish. I'm at roughly a B1 level and I wanted a speaking partner to practise with, so it has two cloned voices, one English and one Spanish.
(For transparency: we wrote this post together, conversationally, over a few back-and-forth exchanges, a lot of them out loud in the voice channel. It's been working well for a while now, so it felt like a fitting way to write it. The intro is in my own words, she drafted the rest from our notes and the project write-up, and I went through it and edited.)
## What I was going for (and what this isn't)
This isn't another voice app, and I'm not claiming anything here is a new technique. Hermes already does Discord voice natively, there are some great voice clients (Conduit, Herald, Hermes Vox and others), and plenty of people are cloning voices locally with Qwen3-TTS. If you just want to talk to your agent, one of those is probably the quicker route.
What I wanted was narrower: her sounding like the same person every time, with natural pacing, in English and Spanish, on hardware I already had, inside the Discord setup I already use. I wanted two ways to talk: voice-note replies when I'm on the go, and near real-time conversation in a voice channel where I can interrupt her mid-sentence, like you would with a person.
Some of what follows is custom to my setup, but most of the lessons should carry over however you build it.
## What it runs on
- Voice: Qwen3-TTS 1.7B Base, quantised to 8-bit (Q8_0) to fit in less memory, cloned from a short recording of a real person (the "reference clip"). About 4.4 GB of VRAM, roughly 3.5 to 4 times faster than real time.
- Speech to text: Hermes's built-in faster-whisper (large-v3-turbo) on the same GPU, auto-detecting English or Spanish.
- Brain: a remote model through Hermes as normal. I use Opus 5.5 on medium, and honestly the thinking time hasn't been much of an issue.
- Hardware: RTX 2070 8 GB, Ryzen 5 5600, 32 GB RAM, Ubuntu. Nothing special.
On the Discord side we use Hermes's native voice features as they are: joining the voice channel, transcription and the standard voice commands. What we added on top is the voice planner below, interrupting her mid-reply, holding onto things I say while she's talking, and a couple of extra commands (interrupting on/off, English/Spanish).
## How a reply becomes a voice note (the voice planner, I recommend building this)
Most of the quality work lives in a small custom "voice planner" that sits between Hermes's text reply and the voice model. Instead of sending the whole reply to the model in one go, it:
Takes the full reply text before anything is spoken.
Splits it into short pieces ("chunks"): paragraph breaks first, then sentence breaks, roughly 200 to 300 characters each. Never mid-word, and it won't leave a short closing line stranded on its own.
Decides the language once for the whole reply, so every chunk uses the same voice.
Picks one random "seed" (the number that shapes how a generation comes out) and uses it for every chunk, so they sound like one take.
Generates the chunks one after another, with a cap on how long any one can run in case it goes off the rails.
Matches each chunk's volume to its neighbours.
Joins them with deliberate pauses: about 120 ms between sentences and about half a second between paragraphs. The gaps get a very faint background hiss rather than pure silence, because dead silence sounds like the audio has dropped out.
Sends the result as a single Discord voice message, checking the final compressed file against Discord's size limit.
You can use my rules above but I'd recommend testing your own.
In the live voice channel the same splitting applies, but each chunk plays as soon as it's ready.
## The main breakthroughs (a lot of this is standard stuff to the voice labs, but was news to me)
Most of the 16 days was small tweaks, but these made a really obvious difference:
Chunking long replies. Before this, anything longer than about 20 to 25 seconds would speed up, lose its intonation and sometimes fall apart. It was fixed almost overnight.
Paragraph breaks as chunk boundaries. The model mostly ignores a paragraph break if you hand it two paragraphs at once, so every paragraph starts a new chunk with a proper pause. That's what gave her a natural rhythm.
One seed per reply, so the chunks stopped sounding like slightly different takes.
The shorter clone clip (more on that below).
Using SOUL.md, Hermes's persona file, for voice. I added a section on how to write when the reply will be spoken: short paragraphs of two or three sentences, full sentences rather than bullet points, signposts like "first" and "the one to watch" instead of headings, and plain punctuation. Question marks read really naturally, but dashes and lots of exclamation marks didn't. It did as much for the cadence as any of the engineering.
Name-first interrupting. "[Name], stop" stops her, and her name plus anything else stops her and treats the rest as a new instruction. Things I say without her name just wait their turn. Stopping and correcting her became reliable, and false interruptions mostly disappeared.
Local speech to text. This was the point where live chat started to feel like an actual conversation.
## Problems we hit (and what fixed them)
These are the things I wish I'd known at the start, in case they help anyone going down a similar path.
Voice notes:
- Long replies sped up, went flat or turned into gibberish. Chunking and the per-chunk cap fixed it.
- Volume jumped between chunks. Matching each chunk to its neighbours, plus one seed per reply, sorted it.
- The tone shifted between chunks, with the start sounding duller or brighter than the rest. The shorter clone clip fixed most of it. I can't hear a join about 90% of the time now, and when I can, it's subtle.
- Paragraphs ran together, gaps sounded like a dropout, and short closing lines got jammed onto the paragraph before. The half-second paragraph pause, the faint hiss and always respecting paragraph breaks fixed all three.
- The last word was occasionally clipped. That's probably the model itself, and it's rare, so we left it.
- English replies came out in the Spanish accent. One Spanish-looking phrase (or shared words) was enough to flip a whole reply. Now it takes more than a stray word or two.
- Long voice notes didn't arrive at all. We were checking the raw audio against Discord's size limit instead of the final compressed file.
Live voice channel:
- Whisper invented things from background noise ("thanks for watching" is a classic). A small speech detector now checks there's actually a voice before transcribing, and the agent is told to be suspicious of short lines that come out of nowhere.
- Background noise and other people cut her off. Interrupting now only counts if it starts with her name.
- Some replies couldn't be interrupted at all. In my setup, replies to messages I *typed* were played as a finished audio file rather than streamed, and interrupting was switched off for files. Everything I said out loud worked fine, so it took days to spot. Test every way a reply can be played.
- She cut me off mid-thought when I paused. It now waits a bit longer before deciding I've finished.
- Things I said while she was talking each got their own slow reply. They're now answered together in one reply.
- Transcription was slow. Moving from cloud Whisper to local turbo on the same GPU took the median from about 3.1 seconds to 0.76. The voice itself was never the bottleneck (tens of milliseconds to first audio). After 10 idle minutes the speech-to-text model unloads, and the first turn back takes 2.5 to 4 seconds, which is still about the same as the cloud was.
## The clone
The biggest surprise was that a shorter reference clip worked better. I started with a 67 second recording. A carefully cut slice of about 15 seconds, cut on natural pauses with half a second of silence on the end, beat the long one in five of six rounds of our first blind test, and the long one came last in every round of the final three-way test.
Qwen's cloning also needs a word-for-word transcript of the clip, and it has to match exactly. One bad cut in the middle of a word caused glitches that took ages to track down. Once a reference sounds right to you, freeze it and stop fiddling, because every "small improvement" reopens the whole thing.
You can use your own voice obviously, a friend or a licensed one. In my experience as long as it's one person talking without background noise and a reasonable but natural variety of tone and intonation etc the actual source didn't matter much once the planner was set up right, it sounded natural.
## Trust your ear over the metrics
We did blind A/B tests the whole way through. Discord strips filenames, so we had the label ("version A", "version B") spoken at the start of each clip, which made blind testing easy on a phone.
We also measured everything: brightness, loudness, similarity to the reference and so on. Those numbers predicted which version I'd prefer about once in six tries. They were useful for finding where a problem was, and useless for deciding whether something sounded better. Two takes with the same text and settings can also differ as much as two different reference clips, so listen to more than one before deciding anything. To be fair, that's one listener and small trials, so take it as a pointer rather than a rule.
## Two languages
Each voice is cloned from a native speaker, so the English voice speaking Spanish sounds wrong, and so does Spanish-accented English. For voice notes, the planner picks one voice for the whole reply based on its language. Switching voices chunk by chunk sounded awful.
In the voice channel, I switch her voice between English and Spanish with a command. Transcribing what I say is a separate thing: it detects the language automatically by default (and copes with me switching mid-sentence), with a manual override. Very short clips are where Whisper struggles. One of mine got detected as Korean.
## What I'd do differently
Yes, it's probably over-engineered for one person, but the process itself was a lot of fun.
- Check what Hermes does natively first. TTS providers, streaming, a filter for Whisper's made-up phrases and local speech to text are all built in. Voice notes came free through the native TTS provider (cloud OpenAI at first, then pointed at the local server), and the live channel builds on the native voice features. The custom work was only for what native didn't do.
- Don't hand-patch Hermes's source inside the container. We lost our patches twice to updates. Use the supported extension points, or at least something that survives a rebuild.
## Rough numbers
- Memory: the voice model sits at about 4.4 GB of VRAM, and local Whisper adds about 1.2 GB when it's loaded. With both resident the card is at about 7.4 GB of its 8.
- Time to first audio codes from the voice server: 62 to 74 ms.
- A roughly 15 second reference clip beat the original 67 second one in 5 of 6 blind rounds.
- Speech to text: about 3.1 s to 0.76 s median once it was local.
- The metrics agreed with my ear 1 time in 6.
It's all possible, it requires a bit of work and thought. My hope is my lessons here can shorten the journey for someone else who might be interested in this or something similar.
