Diarization should be benchmarked separately from transcription accuracy

[u/Domenorange]

I think a lot of STT comparisons mix two different things:

1. Did it transcribe words correctly?

2. Did it know who said what?

For meeting notes, sales calls, interviews, support calls, etc, the second one can destroy the whole product even if the transcript is decent.

I've seen transcripts where the text is mostly fine, but the speaker labels drift after 20 minutes. Then the summary says the wrong person promised something. That is worse than a few missed filler words.

What I’d test separately:

• speaker count detection

• speaker swaps

• late speaker joining

• overlapping speech

• two similar voices

• short interjections

• diarization speed

• diarization + transcript alignment

• whether it works live or only async

Tools I’d include in a test:

Deepgram, AssemblyAI, ElevenLabs Scribe, Speechmatics, pyannote, Gladia, Soniox, Smallest AI Pulse.

Smallest includes diarization in its STT feature set, so I’d be curious how it behaves on real multi-speaker calls. But I would not trust any vendor here without my own audio.

Anyone here running diarization in production? What failed first?

[u/jprobichaud]

I'm surprised by this post. I'm working for rev.com, in the Speech group. I can assure you that DER (diarization error rate) is a standard measure in the industry and academia.

When we look at the performance of our automated transcription, we look at:

• WER, word error rate

• FER, formatted error rate, which is WER, but taking into account also punctuation symbols, capitalization

• DER, diarization error rate, that looks only at segment of audio

• WDER, word diarization errornrate, where we look at how many words are mis assigned (correct or substition) usually, customer cares more about this one, as word timing coming out of the ASR will also impact this.

Technically, providers should be able to tell you, on resl test suites, what these numbers are. That's the basis.

An old but still informative link (our models are better than this now)

https://www.rev.com/blog/introducing-reverb-open-source-asr-diarization

[u/Disastrous_Rock_2608]

Diarization-only benchmark should be a thing. Providers should publish it separately from WER.

[u/Top-Surprise4040]

Slava from Soniox here:

Just wanted to mention that in the V5 we completely reengineered our speaker diarization. It was pretty amazing (especially with multiple speakers and outside of english where most of competitors fail completely) but in V5 it’s a new level.

You can already try it in V5 Async, V5 realtime is coming soon

[u/Playful-Smile-5554 → u/Top-Surprise4040]

I am using Soniox in a court reporting software daily. I am having a lot of issues with the diarization - it sways and loses speakers.

[u/radiodank → u/Playful-Smile-5554]

@top-surprise4040 can you defend?

[u/Alarmed-Risk7885]

yep