Speech data collection · LLM Evaluation
Speech Data Collection for LLM Evaluation
Recruited-speaker speech corpora recorded to a written specification: scripted prompts, spontaneous monologue, or both, with full speaker metadata. Applied to llm evaluation, the specification is driven by one thing: rubric scores with confidence intervals.
- Service
- Speech data collection
- Use case
- LLM Evaluation
- Primary metric
- Rubric scores with confidence intervals
Required data profile
- Native-speaker rater panels per language
- Rubric-based scoring with calibration
- Overlapping assignments for agreement
Technical specification
| Parameter | Standard |
|---|---|
| Sample rate | 48 kHz capture, delivered at 48/16 kHz as required |
| Bit depth | 24-bit capture, 16-bit PCM delivery |
| Format | WAV (PCM), one file per utterance or per session |
| Channels | Mono per speaker; multi-channel on request |
| Noise floor | Studio sessions below -50 dBFS; field sessions specified per project |
| Clipping | Zero tolerance; clipped takes are re-recorded, not repaired |
Process
- Requirement lock: languages, hours, speaker count, demographic quotas, recording conditions
- Prompt design and linguistic review by native reviewers
- Speaker recruitment and screening against quota, with consent capture
- Recording sessions with real-time level and prompt-coverage monitoring
- Automated technical QA on every file (SNR, clipping, duration, silence)
- Native-speaker content QA on a defined sample, escalating to 100% on failure
- Packaging, manifest generation and delivery
Metrics this feeds
- Rubric scores with confidence intervals
- Inter-rater agreement
- Failure-mode distribution
Failure modes to design out
- Raters who are fluent but not native in the variety
- Rubrics written in English and applied to non-English output without localisation
Every file passes automated technical checks. Content QA is sampled at 10% by default and raised per batch when the failure rate crosses the agreed threshold.
Deliverables
- Audio files in the agreed format and naming convention
- Per-utterance manifest (speaker ID, prompt ID, duration, condition)
- Speaker metadata: age band, gender, region, dialect, education band
- Consent records mapped to speaker IDs
- QA report with pass rates and rejection reasons
Frequently asked
Is speech data collection the right service for llm evaluation?
It covers native-speaker rater panels per language. Most llm evaluation programmes combine it with at least one other service; we will say so in the scope rather than selling one line item.
What languages are available?
All 14 languages in the network plus Indian English accent bands.
How is the evaluation set handled?
Collected first, from speakers disjoint from the training cohort, so improvement is measurable.
Scope speech data collection for llm evaluation
Send the metric you need to move.