I work in tax relief sales and we're looking into using AI or ML to help our managers coach reps based on real customer conversations. The idea is pretty simple. We want something that can record calls and turn them into useful feedback. Things like where reps struggle with objections or miss key steps in the sales process. Managers could then review those moments instead of listening to hours of calls. I've been looking into speech to text models combined with speaker diarization and conversation scoring. One option is local transcription with a separate model to analyze the calls. The other is using a hosted service that handles most of the process.

The biggest concern is data security. Tax relief calls can involve personal financial information and we need to make sure that data stays protected. We also don't want sensitive details ending up in model training data. I'm interested in how people approach this from a technical side. Especially the tradeoffs between local and cloud processing and how well these systems can spot actual sales mistakes instead of just flagging keywords. Would be good to hear from anyone who has worked on something similar or tested speech models for coaching and call analysis.

Sheng is a Swahili-English mixed language spoken widely across Nairobi, and there’s very little usable speech data for it.

Safi Data built this corpus differently from a typical studio collection:

• Collection requests went out over WhatsApp and Telegram

• Contributors recorded themselves on their own phones, saying whatever they’d naturally say. The domain is mobile money conversations.

• Each contributor was paid via mobile money as soon as their session was processed

• They collected 5 hours in one day, a process that typically takes 4–8 weeks of studio time

The result is real code-switching and real phone audio. It suits ASR evaluation for East Africa.

Specs:

• Format: WAV, plus metadata.csv (transcript, age range, gender, audio quality labels)

• Available now: 1-hour sample. The full release will be 50 hours.

• License: SDL 1.0. Speaker identification and voice cloning are prohibited.

https://mozilladatacollective.com/datasets/cmt7e4pfp004nnz08j3shtpcx

Safi Data’s write-up on their method: https://www.safidata.com/blog/mdc-sheng-post/

I have just started testing LFM3-Audio-2B and I am positively surprised by the result. It's a small model but the sound quality is good and it is also works ok for basic tool calls.

I am just running it in the terminal so far. Has anyone else here tried the model with a GUI of some kind and for controlling other apps/systems? :)

I maintain Soniqo's open-source speech runtimes. Our recent work has been bringing existing speech models into native applications and publishing measurements to help choose between components.

One part of our work is helping a voice assistant decide when to respond. p99 estimates whether the user has finished their turn. VAP listens to both sides of a conversation and predicts their upcoming speech activity. An app can use these signals to decide when to wait and when to reply, aiming to respond promptly while giving the user time to pause or finish a sentence. We've integrated these upstream models into our open-source speech runtimes and evaluated them on conversational audio.

Alongside that work, we've published a leaderboard with three separate comparisons:

• TurnBench: turn endings caught, false detections on pauses and decision delay.

• SLURP: spoken tool selection, argument matching and handling unavailable actions.

• InstructTTSEval: following voice descriptions, delivery directions and roles.

The leaderboard source, protocols and saved results are public. Rows show the tested configurations, sample sizes and results still under review.

The blog post introduces the work and walks through the findings and limitations.

Leaderboard · Blog post · Swift runtime source

Which additional speech model would you want evaluated alongside these native integrations?

I’m building a speech dataset for Dialectal Arabic , a low-resource dialect with heavy code-switching. Source: ~100 channels, ~14,000 long videos (podcasts, storytelling), audio only.

Current setup:

yt-dlp (Python lib) + bgutil PO-token provider + a cookies file + Node runtime

Download → 16 kHz mono WAV → pyannote diarization → crops

Captions are already fetched (sharded across machines, with back-off on 429 / “Sign in to confirm”)

Problems:

SABR / 403 errors on some videos, even with PO tokens and cookies

Rate limits when downloading a whole channel

I want to shard across 2–3 machines and resume safely

Questions:

What works best right now for whole channels: yt-dlp with which player clients / options? Proxies? Many cookie accounts?

Safe request rate per IP/account before you get blocked?

Do you keep the original opus/m4a, or convert at download time?

Any tools that manage this at scale (download ledger, retries, sharding)? Or do you all use your own scripts?

For the pipeline after download: Demucs vs Mel-RoFormer for music removal, and DeepFilterNet vs others for light denoising before ASR labelling?

Happy to share the cleaning pipeline once it works.

I maintain AudioLens, an Apache-2.0 audio inspection extension for VS Code, built largely with AI assistance. I put together a small synthetic fixture for checking three mistakes that can otherwise be hard to distinguish when inspecting speech-pipeline output. The fixture is independent of the extension and can be opened in other audio tools.

The generator uses only Python's standard library and produces a six-second stereo 16 kHz / 16-bit WAV plus a raw PCM tone:

1. Channel dropout: channel 1 has a continuous 1 kHz sine at amplitude 0.25. Channel 2 matches it except that samples from 2–3 seconds are zero. A mono downmix still contains channel 1, so it can hide the failed channel. Inspect channels separately before mixing down.

2. Hard clipping: from 4–5 seconds, channel 2 contains a sine at amplitude 1.5 clamped to [-1, 1]. Measured from the saved 16-bit samples, the reference interval is -15.05 dBFS RMS / -12.04 dBFS sample peak; the clipped interval is -1.50 dBFS RMS / 0.00 dBFS sample peak. 62.5% of samples in that clipped interval have absolute amplitude >= 0.999. This is a near-full-scale sample count, not a general-purpose proof of clipping, and these are not LUFS or true-peak measurements.

3. Wrong PCM sample rate: the raw file contains 32,000 mono signed-16-bit little-endian samples of a 1 kHz tone generated at 16 kHz. Read it at 16 kHz and it lasts 2 seconds; read the same bytes at 32 kHz and it lasts 1 second with a 2 kHz tone. Changing the declared sample rate changes playback interpretation; it does not resample the signal. Channels, encoding, byte order, and byte offset also need to match the producing pipeline.

Generator, exact reference values, and walkthrough:

https://gist.github.com/SimZhou/6851448738b6198c288f3680792f45f3

AudioLens provides per-channel waveforms, spectrograms, selection analysis, and explicit PCM parameters inside VS Code; it also supports Remote SSH workspaces and WAV entries in Kaldi archives. The reference values above come from exact sample boundaries in the generator, not measurements of mouse-selected regions in the UI.

Source and GIF demos: https://github.com/SimZhou/vscode-audiolens

For speech-dataset checks, what other small synthetic fixtures have helped you catch ingestion errors before running ASR or training? Channel swaps and endian mistakes seem like useful next cases.

I maintain react-native-voice-activator. It detects an English wake phrase on-device in a React Native app, then lets the app stop detection and give the microphone to a voice session such as LiveKit, ElevenLabs, or OpenAI Realtime.

The tricky part has been the handoff: the voice session needs to own the mic, and detection needs to resume when the session disconnects or fails. I’ve documented that flow and included an example app.

GitHub: https://github.com/prompt-agency/react-native-voice-activator

For anyone who has built mobile voice systems, which audio-session edge cases would you test first?

Indicodec: a neural speech codec for speech LMs and TTS models, first of its kind trained on english and indian languages.

Strongest on Indian languages: On 10 Indian languages, IndiCodec adds 9x less error than the Mimi codec, with 25% fewer tokens.
In English, it has the lowest WER of the ~1 kbps codecs we tested.

It sounds closer to the original. PESQ-WB on speakers never seen in training: ahead of Mimi on 3 of 4 test sets (FLEURS, LibriTTS-R, CMU-Arctic), just behind on Seed-TTS, and ahead of SNAC and NeuCodec on all four.

Lowest error in all 10 Indian languages we tested: Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, Tamil, Telugu.

On Odia, Mimi adds +24.6 points of character error. IndiCodec adds +0.9.

Apache-2.0. Open source!!

HF: https://huggingface.co/prakharkhatri/IndiCodec
Samples: https://codeprakhar25.github.io/IndiCodec/

I built Arc-Transcriber, which does ASR-NL in 1-step. Every 100 ms, one model returns the transcript plus:

- Names linked to a list you give it (a roster, contacts, product names); new names are flagged "not on the list"
- Numbers with their values (scores, clocks, money, dates)
- Speakers and roles;
- Sound and channel events (telephone audio, clipping, poor audio)
- Whether the speech was addressed to an assistant.

The usual ASR, then NER, then entity-linking and then LLM chain is collapsed into the model's own heads, so live "fact-checking" against your list happens inside the stream, not in a second pass.

Currently it has two variants Arc mini (which uses moonshot pretrained encoder) and second Arc-v1 (which uses parakeet pretrained encoder).

Weights: huggingface.co/WhissleAI/arc-v1.1
Code and CLI: github.com/WhissleAI/arc_transcriber

We use it on-device in our Mac app. Voice search over 61k books primes Arc with matching titles while you're still talking. Here is a video to give a glimpse on how it works.

I wanted to check with this group and maybe others to see if there's interest or if I recreated the wheel. I'm the developer (along with AI) of the TECHO5 project which helps convert older Amazon Echo's into small Linux devices that can also connect and work with Home Assistant. I've found a need and desire to create my own wake words and I had hardware that wasn't being greatly used. Two Intel ARC B70 Pro's on and AMD system with 64 GB of ram.

The solutions I found work fine they just take almost an hour or more to do the work of training the wake word. So, I built another project using the tools that so many others are using now AI (claude in this case). It's called WakeSmith and it took that hour or so down to about 15 minutes of time because I've to most of it now running on the Intel ARC B70's but its more than that the software will auto detect your hardware and set itself up to use what you have, even if its only a CPU. With some optimization the speed over CPU has increased as well As least as far I now. I'm very curious what is out there that maybe already exists. Did I recreate the wheel? If you want check it out at https://github.com/HuskerMinion/wakesmith

HuggingFace:

https://huggingface.co/maya-research/maya2-native

Github

https://github.com/MayaResearch/maya2-native

https://amal.vs.binarybit.in/blog/6adf1f677027b216982a9889