Evaluation
How do you build an evaluation set for speaker diarisation?
Updated 2026-08-01 · 4 min read
Short answer
Build the evaluation set from data recorded under deployment conditions by speakers who never appear in training, then freeze it. For speaker diarisation, that means sampling per-speaker isolated channels with a mixed reference, genuine overlap preserved across the dialects, devices and noise environments your users actually have, annotating to a stricter standard than the training set, and reporting diarisation error rate, overlap detection recall per slice rather than as one aggregate. A 5–20 hour eval set built this way is worth more than a hundred hours of training data, because it is the only thing that tells you whether the rest worked.
Key takeaways
- Speaker overlap between train and eval is the fastest way to fool yourself.
- Report per-slice metrics — aggregate numbers hide the dialect or device where the model fails.
- Freeze the eval set and version it; a moving benchmark measures nothing.
Sampling rules
- Zero speaker overlap with any training batch
- Recorded on the devices and channels of the deployment, not in a studio
- Slices sized so per-slice metrics are statistically meaningful
- Include the hard cases deliberately: noise, overlap, accented speech, rare entities
- Collected after the training spec is frozen, so it cannot be gamed
What to measure
Report each metric per slice — language, dialect, device, noise condition, speaker age band. An aggregate score that improves while a slice regresses is a launch risk disguised as progress.
- Diarisation error rate
- Overlap detection recall
- Speaker-count accuracy
Annotation standard
Evaluation annotation should be stricter than training annotation: double-blind transcription with adjudication of disagreements, and a written arbitration rule. Where two annotators disagree on a reference, the metric computed against it is noise.
For speaker diarisation, the specific pitfalls to watch are: overlap edited out during recording, single-channel-only capture leaving no reliable ground truth
Sizing the set
| Purpose | Typical size | Refresh |
|---|---|---|
| Smoke test in CI | 1–2 hours | Stable, rarely changed |
| Release gate | 5–10 hours | Refreshed annually |
| Slice diagnostics | 10–20 hours | Extended as new markets launch |
Keeping it honest over time
Once a benchmark has been optimised against for a year, its numbers stop predicting production performance. Plan a rotation: keep the frozen set for comparability and add a fresh unpublished set each year for the real read.
Frequently asked questions
How large should an evaluation set be?
5–20 hours for most speech use cases, sized so each reported slice has enough utterances to be meaningful rather than so the total looks impressive.
Can training data be reused for evaluation?
No. Any speaker, session or recording condition shared with training inflates the score in ways that do not survive deployment.
Who should annotate the evaluation set?
Native-speaker annotators who did not work on the training corpus, working double-blind with adjudication of disagreements.
How often should it be refreshed?
Keep the frozen set for comparability and add a fresh, unpublished set annually so you can detect benchmark overfitting.
Related reading
Turn this into a dataset specification
Tell us the languages, speaker count and minutes. You get a written scope, a protocol and a fixed price within one working day.