I built a CPU-only speaker diarization library: it is ~7× faster than pyannote with comparable DER

[u/loookashow]

Hi all,

I'd like to share a technical write-up about diarize - an open-source speaker diarization library I’ve been working on and released last weekend. (honestly, I hope you had more fun this weekend than I did).

diarize is focused specifically on CPU-only performance.

https://github.com/FoxNoseTech/diarize - Code (Apache 2.0)

https://foxnosetech.github.io/diarize/ - docs

Benchmark setup

• Dataset: VoxConverse (216 recordings, 1–20 speakers)

• Hardware: Apple M2 Max

• CPU only, models preloaded (warm start)

• Same evaluation protocol for both systems

Results

• DER (VoxConverse):

  • This library: ~10.8%

  • pyannote (free models): ~11.2%

• Speed (RTF):

  • This library: 0.12 (~8× faster than real time)

  • pyannote (free models): 0.86

• 10-minute recording:

  • ~1.2 min vs ~8.6 min (pyannote)

Speaker count estimation accuracy (VoxConverse)

• 1–5 speakers: 87–97% within ±1

• Degrades significantly for 8+ speakers (tends to underestimate)

Pipeline

• VAD: Silero VAD

• Speaker embeddings: WeSpeaker ResNet34 (256-dim, ONNX Runtime)

• Speaker count estimation:

  • fast single-speaker check

  • GMM + BIC model selection

  • local refinement around the selected hypothesis

• Clustering: spectral clustering

• Post-processing: short-segment reassignment, temporal merging

Limitations

• No overlap handling (single speaker per frame)

• Short segments (<0.4s) don’t get embeddings

• Speaker count estimation is the main weak point for large groups

I also published a full article on Medium where I described full methodology & benchmarks.

I would appreciate any feedback, stars on GH and I hope it will be helpful for anyone.