Service explainers
What is ai voice evaluation and how does it work?
Updated 2026-08-01 · 4 min read
Short answer
Human evaluation of your speech models: MOS and preference testing for TTS, WER-in-context review for ASR, and native-speaker judgement on naturalness and intelligibility. In practice the work runs as protocol design and sample-size calculation, then panel recruitment and calibration on reference items, then blind evaluation with attention checks, and you receive raw per-rater scores, aggregated results with confidence intervals. 1-3 weeks per evaluation round.
Key takeaways
- Attention checks and reference anchors are embedded so unreliable raters are detected and excluded before analysis.
- Typical buyers: TTS QA, ASR benchmarking, Model release gating.
- Recruitment approach: Raters are excluded if they contributed to the training data for the model under test.
What the service covers
Human evaluation of your speech models: MOS and preference testing for TTS, WER-in-context review for ASR, and native-speaker judgement on naturalness and intelligibility.
- Raw per-rater scores
- Aggregated results with confidence intervals
- Error-type analysis
- Recommended fix priorities
Technical specification
These are defaults, not limits. Where your pipeline requires different values, they replace ours in the statement of work rather than being converted after delivery.
| Parameter | Standard |
|---|---|
| TTS | MOS (1-5), MUSHRA, and A/B preference protocols |
| ASR | Error typing: substitution, deletion, insertion, code-switch failure |
| Panel | Native speakers of the target variety, screened and calibrated |
| Sample size | Powered per the effect size you need to detect |
| Reporting | Per-item scores plus aggregate with confidence intervals |
How the work runs
- Protocol design and sample-size calculation
- Panel recruitment and calibration on reference items
- Blind evaluation with attention checks
- Statistical analysis
- Report with per-error-type breakdown
Quality control and acceptance
Attention checks and reference anchors are embedded so unreliable raters are detected and excluded before analysis.
Failures are remedied by re-collection rather than by editing delivered files, because repaired audio carries artefacts that survive into the trained model.
Who this is for
Recruitment for this service works as follows. Raters are excluded if they contributed to the training data for the model under test.
- TTS QA
- ASR benchmarking
- Model release gating
Timelines
1-3 weeks per evaluation round.
Multi-language programmes run in parallel rather than in sequence, so a five-language scope does not take five times as long.
Frequently asked questions
What is included in ai voice evaluation?
Raw per-rater scores, Aggregated results with confidence intervals, Error-type analysis, delivered against a written specification with acceptance criteria attached.
How long does ai voice evaluation take?
1-3 weeks per evaluation round.
How is quality measured?
Attention checks and reference anchors are embedded so unreliable raters are detected and excluded before analysis.
Which languages are supported?
Fifteen Indian languages plus Indian English, including Hindi, Marathi, Tamil, Telugu, Kannada.
Related reading
Turn this into a dataset specification
Tell us the languages, speaker count and minutes. You get a written scope, a protocol and a fixed price within one working day.