Evaluation
How do you build an evaluation set for machine translation?
Updated 2026-08-01 · 4 min read
Short answer
Build the evaluation set from data recorded under deployment conditions by speakers who never appear in training, then freeze it. For machine translation, that means sampling sentence-aligned parallel corpora, register-matched to your product across the dialects, devices and noise environments your users actually have, annotating to a stricter standard than the training set, and reporting human adequacy and fluency scores, terminology compliance rate per slice rather than as one aggregate. A 5–20 hour eval set built this way is worth more than a hundred hours of training data, because it is the only thing that tells you whether the rest worked.
Key takeaways
- Speaker overlap between train and eval is the fastest way to fool yourself.
- Report per-slice metrics — aggregate numbers hide the dialect or device where the model fails.
- Freeze the eval set and version it; a moving benchmark measures nothing.
Sampling rules
- Zero speaker overlap with any training batch
- Recorded on the devices and channels of the deployment, not in a studio
- Slices sized so per-slice metrics are statistically meaningful
- Include the hard cases deliberately: noise, overlap, accented speech, rare entities
- Collected after the training spec is frozen, so it cannot be gamed
What to measure
Report each metric per slice — language, dialect, device, noise condition, speaker age band. An aggregate score that improves while a slice regresses is a launch risk disguised as progress.
- Human adequacy and fluency scores
- Terminology compliance rate
- Back-translation divergence
Annotation standard
Evaluation annotation should be stricter than training annotation: double-blind transcription with adjudication of disagreements, and a written arbitration rule. Where two annotators disagree on a reference, the metric computed against it is noise.
For machine translation, the specific pitfalls to watch are: pivoting everything through english, post-edited machine output passed off as human translation
Sizing the set
| Purpose | Typical size | Refresh |
|---|---|---|
| Smoke test in CI | 1–2 hours | Stable, rarely changed |
| Release gate | 5–10 hours | Refreshed annually |
| Slice diagnostics | 10–20 hours | Extended as new markets launch |
Keeping it honest over time
Once a benchmark has been optimised against for a year, its numbers stop predicting production performance. Plan a rotation: keep the frozen set for comparability and add a fresh unpublished set each year for the real read.
Frequently asked questions
How large should an evaluation set be?
5–20 hours for most speech use cases, sized so each reported slice has enough utterances to be meaningful rather than so the total looks impressive.
Can training data be reused for evaluation?
No. Any speaker, session or recording condition shared with training inflates the score in ways that do not survive deployment.
Who should annotate the evaluation set?
Native-speaker annotators who did not work on the training corpus, working double-blind with adjudication of disagreements.
How often should it be refreshed?
Keep the frozen set for comparability and add a fresh, unpublished set annually so you can detect benchmark overfitting.
Related reading
Turn this into a dataset specification
Tell us the languages, speaker count and minutes. You get a written scope, a protocol and a fixed price within one working day.