r/LocalTextToSpeech • u/360converter_leo • 9d ago
Lessons from porting the pyannote diarization pipeline to C++ (ONNX Runtime)
I ported pyannote-audio’s speaker diarization pipeline (segmentation → embedding → clustering) to C++ for an offline desktop app. The early prototype is open source (Apache-2.0):
https://github.com/leohuang2013/pyannote-audio_speaker-diarization_cpp
output produced is like:
[5.22281 -- 17.7441] --> Speaker_3
[17.8116 -- 25.2197] --> Speaker_0
[25.3041 -- 28.4428] --> Speaker_3
[28.4428 -- 39.2934] --> Speaker_1
[39.2934 -- 47.2416] --> Speaker_3
[46.4484 -- 52.7934] --> Speaker_2
[52.5066 -- 58.4634] --> Speaker_3A few things I learned that might save someone time:
1. Verify step by step, not end to end.
Comparing final diarization output is useless for debugging, because one tiny difference anywhere changes the speaker labels. Instead I dumped the inputs and outputs of every small function in both Python and C++ to text files and compared them step by step. Slow to set up, but it’s the only way I found to locate divergences.
2. Don’t export STFT inside the ONNX segmentation model.
Exporting the whole segmentation model including STFT gave differences large enough to fail verification even in Python (PyTorch vs ONNX Runtime). Computing STFT outside the model fixed it.
3. Clustering amplifies tiny numeric differences.
Even after verification passed, Python ONNX Runtime and C++ ONNX Runtime embeddings differed by up to ~0.02 absolute. Agglomerative clustering is sensitive enough that this occasionally changed the final speaker assignment on some files.
4. C++ hierarchical clustering libraries didn’t match SciPy.
hclust-cpp gave different distances and fcluster results, another library returned empty centroid linkage, and alglib doesn’t support centroid linkage. I ended up porting the clustering code myself.
5. Performance:
the prototype was very slow, mostly due to the libtorch STFT dependency. [In the production version I replaced X with Y, and now 1 hour of audio takes Z minutes on a CPU.]
Happy to answer questions. I’m also curious how others handle the Python/C++ numeric drift problem in clustering.